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Abstract 

The  Air  Force  Materiel  Command’s  mission  is  to  acquire  and  maintain  Air  Force 
weapon  systems.  The  primary  contracting  method  of  acquiring  weapon  systems  uses 
source  selections.  A  source  selection  is  a  means  of  evaluating  proposals  in  terms  of 
credibility,  performance,  and  cost.  Among  the  different  criteria  used  in  this  selection, 
recent  emphasis  has  been  placed  on  the  use  of  past  performance. 

Formal  guidelines  for  collecting  and  storing  past  performance  information  have 
already  been  established  for  acquisitions  over  $5  million.  However,  recent  policy  has 
directed  the  use  of  past  performance  for  all  acquisitions  over  $100,000.  Presently,  the 
$100,000  to  $5  million  range  leaves  contracting  officers  and  buyers  great  flexibility  and 
little  guidance  in  utilizing  past  performance  for  their  source  selections.  The  purpose  of 
this  exploratory  study  is  to  demonstrate  how  these  professionals  are  responding  to  the 
past  performance  emphasis  within  this  acquisition  range.  Methods  of  collecting,  storing, 
and  protecting  past  performance  information  were  obtained  as  well  as  the  kinds  of 
information.  How  they  determine  the  information’s  relevancy  was  also  outlined.  In 
addition,  this  research  effort  probed  lessons  learned  by  these  acquisition  professionals. 
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AN  EXAMINATION  OF  THE  PAST  PERFORMANCE  PROCEDURES 


USED  IN  AIR  FORCE  MATERIEL  COMMAND  SOURCE  SELECTIONS 


I.  Introduction 


General  Issue 

The  Air  Force  Materiel  Command’s  mission  is  to  contribute  to  combat 
superiority,  readiness,  and  sustainability.  The  AFMC  Mission  states,  “Through 
integrated  management  of  research,  development,  test  acquisition,  and  support,  we 
advance  and  use  technology  to  acquire  and  sustain  superior  systems  in  partnership  with 
our  customers  and  suppliers”  (AFMC  Home-page,  1997:1). 

Acquisition  of  such  technology  involves  the  purchase  of  unique  requirements.  A 
sealed-bidding  procurement  implies  that  the  only  evaluation  factor  necessary  is  price. 
However,  AFMC  does  not  deal  primarily  with  commercial-off-the-shelf  items,  which  are 
ideal  for  sealed  bids.  There  is  usually  more  than  one  method  to  deliver  the  supplies  and 
services  of  some  of  AFMC’s  unique  requirements.  Therefore,  competitive  proposals  may 
be  more  suitable  for  the  acquisition.  Here,  offerors  are  given  objectives  and  in  return  they 
provide  a  proposal  on  how  they  are  going  to  meet  them.  The  Government’s  evaluation  of 
these  proposals  should  be  evaluated  on  factors  besides  price  alone.  For  large 
acquisitions,  this  method  of  evaluation  is  termed  the  “source  selection.”  It  is  a  complex 
and  thorough  process  of  procuring  systems  by  means  of  evaluating  competitive  proposals 
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in  terms  of  achieving  the  best  expected  value.  According  to  the  Air  Force  Federal 
Acquisition  Regulation  Supplement , 

the  objective  of  the  source  selection  process  is  to  select  the  source  whose  proposal 
has  the  highest  degree  of  credibility  and  whose  performance  can  be  expected  to 
best  meet  the  Government’s  requirements  at  an  affordable  cost.  (1996:BB-1) 

In  achieving  best  value,  a  philosophy  exists  that  contractors  will  work  in  the 

future  as  they  have  demonstrated  in  the  past.  The  past  performance  evaluation  process  is 

a  mechanism  that  provides  a  certain  confidence  that  the  offeror  has  the  capability  and 

desire  to  accomplish  the  promises  of  the  proposal.  Formerly,  performance  risk  ratings 

have  been  insignificant  in  the  final  award  decision  because  responsibility  determinations 

already  served  as  an  indicator  that  the  offeror  can  successfully  accomplish  the  contract 

requirements.  This  determination  is  made  when  the  contracting  officer  must  decide  if  the 

apparent  successful  offeror  can  perform  the  contract.  The  past  performance  criterion 

differs  from  a  responsibility  determination  in  that  it  offers  a  comparison  of  an  offeror’s 

track  record  with  other  offerors  during  the  competitive  selection  process  to  determine 

which  one  has  the  best  value.  The  responsibility  determination  merely  ensures  that  the 

contractor  has  had  a  satisfactory  performance  record  in  order  to  do  business  with  the 

Government.  Using  past  performance  information  also  assists  in  the  development  of  an 

acquisition  strategy  (e.  g.,  contract  type),  places  more  effective  management  attention  on 

contractor  performance,  and  even  improves  contractor  performance. 

Using  past  performance  evaluations  is  not  new  to  Government  source  selections. 

However,  in  the  stream  of  Acquisition  Reform,  the  Office  of  Federal  Procurement  Policy 

officially  created  past  performance  as  an  evaluation  criterion  in  1993.  In  1995,  under  one 
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of  the  “Lightning  Bolt  Initiatives,”  the  Air  Force  formed  an  investigative  action  team  to 
manage  effective  implementation  of  past  performance  policy. 

Either  a  Performance  Risk  Assessment  Group  (PRAG)  or  a  smaller  unit,  which 
includes  a  contracting  officer  and  one  or  more  buyers,  evaluates  past  performance.  It  is 
assessed  in  terms  of  the  offeror’s  ability  to  accomplish  the  work  detailed  in  the  proposal, 
given  the  offeror’s  past  and  present  work  record.  According  to  the  Air  Force  Federal 
Acquisition  Regulation  Supplement,  “past  and  present  performance  must  demonstrate  an 
ability  to  achieve  management  objectives  shown  by  performance  on  related  Government 
efforts”  (1996:  BB-44).  Using  past  performance  information  demonstrates  an  “actual 
ability  to  perform”  and  increased  use  of  this  evaluation  criteria  should  be  widely 
supported  by  the  defense  industry  (Brislawn  and  Dowd,  1996:16). 

Specific  Problem 

For  acquisitions  over  $5  million,  PRAG’s  have  regulatory  guidelines  for 
obtaining  and  storing  past  performance  information  (i.  e.,  AFFARS  Appendix  AA  and 
BB  and  AFMC  Instruction  64-107,  Contractor  Performance  Assessment  Reporting 
System).  However,  acquisitions  under  $5  million  do  not  require  the  use  of  PRAG’s  for 
performance  risk  ratings.  The  contracting  officer  and  buyer(s)  have  great  flexibility  in 
how  they  perform  this  function  but  they  do  not  have  any  regulatory  guidance  on  how. 
Thus,  in-house  methods  have  been  established  to  address  past  performance.  Policy 
officials  at  HQ  AFMC  wish  to  provide  guidance  to  enhance  a  more  effective  practice  of 
using  past  performance  as  a  criteria  in  source  selections.  The  results  of  such  guidance 
could  offer  a  standard  process  which  promotes  equality  and  fairness  to  offerors.  A 
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systemic,  exploratory  study  is  necessary  to  learn  how  the  contracting  officers  and  buyers 
are  currently  implementing  the  past  performance  process  in  order  to  provide  HQ  AFMC 
feedback  for  their  guidance.  This  research  effort  focuses  on  the  past  performance 
information  system  for  acquisitions  between  $1  to  5  million  and  makes  recommendations 
for  effective  practice  of  the  past  performance  process.  Implementation  below  $1  million 
is  highly  unexpected. 

Investigative  Questions 

This  research  will  investigate  how  AFMC  acquisition  professionals  have  been 
implementing  past  performance  in  their  source  selections.  The  research  effort  is  broken 
into  the  following  investigative  questions: 

1)  How  are  the  Contracting  Officers  storing  their  past  performance  information? 

2)  How  are  the  Contracting  Officers  protecting  the  past  performance 
information? 

3)  How  are  the  Contracting  Officers  making  their  relevancy  determinations? 

4)  What  sources  of  collecting  past  performance  information  are  being  used? 

5)  What  questions  are  being  asked  when  collecting  the  information? 

6)  What  similarities  and  differences  exist  between  past  performance  information 
systems? 

When  integrated  together,  the  answers  to  these  questions  will  reveal  the  past 
performance  evaluation  procedures  currently  used  for  the  lower  dollar  acquisitions.  As  a 
result,  an  outline  of  the  commonalities  and  differences  among  these  methods  will  be 
provided.  Then,  an  analysis  will  investigate  the  similarities  and  differences  between  past 
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performance  information  systems.  From  this  analysis,  suggestions  will  be  made  for 
future,  lower  dollar  AFMC  acquisitions. 


II.  Literature  Review 


Introduction 

This  chapter  describes  the  top-level  emphasis  for  using  past  performance  criteria. 
Research  involving  evaluation  criteria  is  then  presented.  Next,  an  overview  of  the  source 
selection  process  is  provided  to  gain  an  understanding  of  how  past  performance  may  be  a 
significant  evaluation  criterion  in  the  procurement  of  best  value  contracts.  Performance 
risk  assessment  is  then  addressed  to  explain  how  past  performance  is  actually  used  in  the 
source  selection.  Sources  of  past  performance  information  are  listed  and  finally  the 
relevancy  determination  is  also  addressed. 

Recent  Emphasis  on  Past  Performance 

The  use  of  past  performance  information  was  given  a  boost  in  1986  in  a  report  to 
President  Reagan  entitled  A  Formula  for  Action ,  A  Report  to  the  President  on  Defense 
Acquisition  by  the  President’s  Blue  Ribbon  Commission  on  Defense  Management.  This 
report  is  more  commonly  called  “The  Packard  Commission”  and  calls  for  more  use  of 
commercial-style  practices  in  Government  acquisitions.  The  doctrine  was  captured  in  the 
following  passage: 

Typically,  an  industrial  company  will  keep  lists  of  qualified  suppliers  that  have 
maintained  historically  high  standards  of  product  quality  and  reliability.  As  long 
as  these  standards  are  maintained,  industrial  buyers  do  not  require  exhaustive 
inspection,  and  thereby  save  expense  on  both  sides.  Suppliers  are  highly 
motivated  to  get  and  stay  on  lists  of  qualified  suppliers  by  consistently  exceeding 
quality  control  standards.  (1986:62-63) 

As  the  Packard  Commission  emphasized  quality  and  establishing  performance 
measures  for  the  basis  of  contract  award,  the  General  Accounting  Office  (GAO) 
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performed  a  study  from  1988  to  1992  on  contracts  awarded  by  using  past  performance  as 
an  award  criteria  rather  than  simply  selecting  the  lowest  priced  offeror.  The  vendors’  past 
performance  information  related  experience,  quality,  and  delivery.  The  pilot  program 
identified  an  insignificant  increase  in  price  when  awarding  to  a  member  of  the  “Quality 
Vendor  Programs”  (1993:5). 

In  1993,  the  Federal  Register  published  its  implementation  of  OFPP  Policy  Letter 
92-5  to  make  past  performance  an  official  evaluation  criterion  in  source  selections.  Past 
performance  information  has  always  been  used  in  responsibility  determinations,  but  this 
policy  letter  establishes  past  performance  as  a  specific  evaluation  criteria  to  be  used  in 
equal  comparison  with  the  factor  assessment  and  proposal  risk  for  each  factor 
(1993:3573). 

Formerly,  performance  risk  ratings  have  been  insignificant  to  the  final  award 
decision  because  responsibility  determinations  already  serve  as  an  indicator  that  the 
offeror  can  successfully  accomplish  the  contract  requirements.  This  determination  is 
made  after  the  apparent  successful  bidder  is  identified  and  when  the  contracting  officer 
must  decide  if  the  offeror  can  successfully  perform  the  contract.  The  past  performance 
criterion  differs  from  a  responsibility  determination  in  that  the  past  performance  criterion 
offers  a  comparison  of  an  offeror’s  performance  track  record  with  other  offerors  during 
the  competitive  selection  process  to  determine  which  one  has  the  best  value.  The 
performance  track  record  includes  information  such  as  the  “number  and  severity  of  an 
offeror's  problems,  the  effectiveness  of  corrective  actions  taken,  [and]  the  offeror's  overall 
work  record”  (DoD  Wisdom  Focal  Point,  1996:1).  In  contrast,  responsibility  considers 
having  the  financial  resources,  satisfactory  performance  record,  business  integrity  and 
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ethics,  organization,  experience,  skills,  controls,  equipment,  and  facilities  to  successfully 
perform  the  contract  and  comply  with  the  delivery  schedule  (FAR  9.104).  Besides  using 
past  performance  information  as  a  competition  factor,  it  also  assists  in  the  development  of 
an  acquisition  strategy,  places  more  effective  oversight  on  contractor  performance,  and 
even  improves  contractor  performance  (Little,  1996:38). 

From  the  direction  of  OFPP  Policy  Letter  92-5,  the  change  to  acquisition  policy 
requires  Government  agencies  “to  prepare  evaluations  of  contractor  performance  on  all 
contracts  over  $100,000  and  to  specify  past  performance  as  an  evaluation  factor  in 
solicitations  for  offers  for  all  competitively  negotiated  contracts  expected  to  exceed 
$100,000”  (1993:3573).  The  letter  also  recognizes  that  a  contractor’s  past  performance 
information  may  reflect  poor  subcontractor  performance.  In  this  regard,  the  policy 
dictates  that  “A  prime  contractor’s  ability  to  select  subcontractors  that  perform  is 
indicative  of  the  contractor’s  management  ability”  (1993:3574). 

The  Federal  Acquisition  Streamlining  Act  of  1994  made  past  performance  a 
mandatory  source  selection  evaluation  factor  (Scott,  1995:4).  In  response,  twenty  federal 
departments  and  agencies  participated  in  the  past  performance  pledge  on  January  26, 

1994  in  the  award  of  60  contracts  (ARNet,  1996:5).  The  list  of  pledges  included  the 
Department  of  Air  Force  and  Defense  Logistics  Agency.  All  pledges  supported  past 
performance  as  a  major  selection  criteria  by  establishing  interagency  teams  for  effective 
structuring  of  solicitations  and  evaluation  offers,  identifying  and  eliminating  obstacles, 
and  providing  information  and  lessons  learned  to  the  others  (ARNet,  1996:Appendix  1 :5). 
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The  Federal  Acquisition  Regulation  implements  OFPP  Policy  Letter  92-5  under 
FAR  42. 1 5.  Its  guidance  suggests  that  agencies  shall  provide  inputs  into  evaluation 
procedures  (FAR  42.1503(a)). 

The  Air  Force  responded  to  Acquisition  Reform  with  the  issuance  of  nine 
Lightning  Bolt  Initiatives.  Lightning  Bolt  #6,  which  states  “Enhance  the  role  of  past 
performance  in  source  selections,”  responds  to  the  FAR’s  direction  that  the  Air  Force 
shall  provide  guidance  into  evaluation  procedures  (Dept  of  Air  Force  “Air  Force 
Acquisition,”  1995:2).  Based  on  this  Lightning  Bolt,  a  team  led  by  HQ  AFMC 
implemented  past  performance  policy  revisions  to  the  AFFARS  Supplement,  provided 
standards  and  methods  to  assess  past  performance,  and  improved  the  effectiveness  of  the 
Contractor  Performance  Assessment  Reporting  System  by  revising  the  instructions  and 
forms.  However,  the  guidance  provided  in  AFFARS  Appendices  AA  and  BB  and 
CPARS  are  not  applicable  for  acquisitions  under  $5  million. 

In  a  report  to  the  Office  of  the  Secretary  of  Defense,  Arthur  D.  Little,  Inc.,  a 
research  company,  issued  seven  recommendations  for  the  formulation  of  a  past 
performance  policy  (Little,  1996:10-1 1).  The  recommendations  included  that 
information  systems  should  be  decentralized  to  the  service  level  instead  of  a  defense-wide 
system  because  the  value  of  past  performance  decreases  as  a  past  performance  system  is 
standardized.  Thus,  focus  should  be  placed  on  the  past  performance  information  that  is 
relevant  to  each  acquisition’s  business  area.  Little  also  notes  the  current  view  is  fixed  on 
one  facet  (collection  and  validation)  of  past  performance;  it  should  be  placed  on  the  total 
program  context  instead.  A  simple  approach  based  on  user  needs  for  a  system  that 
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enables  information  sharing  will  exert  a  horizontal  integration  effect  “by  joining  similar 
business  areas  across  the  services  and  DLA”  (1996:1 1). 

In  the  same  report  to  the  OSD,  the  Little  report  outlines  the  importance  of  using 
past  performance  information  in  Department  of  Defense  source  selections.  This  outline 
contains  the  following  six  reasons  why  past  performance  evaluations  are  useful:  1)  they 
evaluate  performance  risk;  2)  they  assist  the  development  of  an  acquisition  strategy  (e.  g., 
contract  type);  3)  they  require  more  effective  management  of  contractor  performance;  4) 
they  improve  contractor  performance;  5)  they  create  more  effective  allocation/oversight; 
and  6)  they  streamline  the  technical  and  management  evaluations  with  the  element  of 
performance  risk  (Little,  1996:38). 

A  schedule  for  the  implementation  of  past  performance  policies  was  outlined  by 
Dr.  Kelman  (OFPP  Administrator  of  Federal  Procurement  Policy).  The  schedule  required 
the  use  of  past  performance  as  a  source  selection  factor  for  all  solicitations  over  $1 
million  by  July  1,  1995,  solicitations  over  $500,000  by  July  1,  1997,  and  solicitations 
over  $100,000  by  January  1,  1999  (ARNet,  1996:10).  These  mandatory  milestones  are 
established  in  FAR  15.605(b)(l)(ii).  Dr.  Kelman  further  details  the  substantiation  of  a 
performance  reporting  system  for  contracts  over  $1  million  by  July  1,  1995;  contracts 
over  $500,000  by  July  1,  1996;  and  contracts  over  $100,000  by  January  1,  1998  (ARNet, 
1996:10). 
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Best  Value  and  Past  Performance 


As  stewards  of  the  public’s  money,  the  Government  is  compelled  to  find  the  most 
effective  and  efficient  decision  criteria  for  awarding  contracts.  The  most  common  criteria 
is  using  the  lowest  priced  offer.  However,  there  may  be  a  reason  why  the  offer  is  priced 
so  low,  as  the  axiom  suggests  “you  get  what  you  pay  for.”  Therefore,  the  Government  is 
trying  to  develop  a  standardized  concept  of  achieving  the  best  expected  value. 

Attempts  have  been  made  to  explain  the  trade-off  relationship  with  cost  and  all 
other  factors,  or  criteria,  that  would  account  for  the  “best  value.”  The  “beef  stew” 
metaphor  identifies  ingredients  that  combine  into  best  value,  such  as  trading  cost  for 
technical  benefits  (Solloway,  1989:9).  In  this  decision-making  image,  all  relative  criteria 
are  added  together  in  a  relative  ranking  of  importance  and  then  reflected  in  a  final 
selection  decision.  The  ranking  of  evaluation  criteria  changes  depending  on  the  particular 
buying  situation.  Reliability  and  maintainability  may  be  added  as  evaluation  factors  to 
the  list  of  more  standard  factors  such  as  cost,  technical,  and  performance  (Hamen,  1985). 

Best  value  can  be  described  as  the  approach  that  “gives  the  government  greater 
flexibility  and  discretion  to  make  trade-offs  between  cost/price  and  quality  factors  such  as 
technical  approach,  management  capabilities,  and  past  performance”  (Scott,l  995:3).  It  is 
a  practice  used  by  both  the  public  and  private  business  sectors.  The  difference  is  that  best 
value  criteria  increases  the  risk  of  protest  because  it  involves  a  subjective  judgment  while 
creating  an  environment  in  which  all  offerors  have  equal  basis  for  competition  (Scott, 
1995:3). 

Case-based  decision  theory  (CBDT),  an  economic  decision  making  theory  which 
provides  maximum  utility  based  on  memories  of  the  past  when  faced  with  an  infinite 
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range  of  uncertain  outcomes,  supports  the  use  of  past  performance  information.  CBDT  is 
more  simply  stated  that  past  performance  is  a  driver  in  present  decisions  (Gilboa  and 
Schmeidler,  1995:608).  CBDT  was  introduced  as  a  way  to  delineate  the  uncertainty  of 
the  future  into  a  resemblance  of  what  has  occurred  in  the  past.  In  that  respect,  past 
performance  information  provides  a  source  selection  decision  an  assessment  of  each 
offeror’s  performance  risk.  Gilboa  and  Schmeidler  recommend  application  of  CBDT  in 
consumer  purchases  based  on  firm  reputations  (1995:627, 628). 

The  trade-off  relationship  was  also  categorized  into  10  combinations  of  buying 
situations  with  the  most  common  criteria  for  each  situation  identified  (Lehmann,  1982). 
Past  performance  criteria  (referred  to  as  “adaptive  criteria”)  was  ranked  most  common 
among  standard,  simple,  high-dollar-commitment  buying  situations  (1982:1 1).  Also,  the 
collection  of  past  performance  information  should  be  specifically  tailored  to  each  type  of 
acquisition  (Brislawn  and  Dowd,  1996:18). 

Past  performance  information  has  demonstrated  its  credibility  as  an  evaluation 
factor  in  achieving  best  value.  In  a  study  of  300  major  Air  Force  source  selections,  all 
but  two  resulted  in  actual  contract  performance  the  same  as  that  predicted  by  previous 
past  performance  determinations  (Norton,  1986:x). 

The  emphasis  of  past  performance  is  visible  in  both  the  public  and  private  sector 
of  business  (Sheth  et  al,  1983:8).  However,  the  use  of  past  performance  information 
within  industry  is  not  comparable  to  that  of  the  Government’s  size,  scope  and  complexity 
(Little,  1996:5).  However,  industry  exhibits  some  of  the  same  past  performance  trends  as 
the  Government,  as  Little  identifies  the  following  commonalities  in  use  of  past 
performance  information: 
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business  area  strategy  and  management,  supplier  approvals,  quality  system 
assessments,  performance  assessments,  performance  measurement  feedback, 
supplier  development  and  improvement  initiatives,  total  cost  assessments,  item 
and  part-level  certifications,  and  recognition  programs.  (1996:6) 

Therefore,  it  is  possible  to  look  to  commercial  practices  for  guidance  in  implementing  the 

use  of  past  performance  information  in  Government  procurements. 

Although  the  use  of  past  performance  information  is  beneficial  to  achieving  best 

t 

value,  its  validity  may  be  challenged.  Dr.  Kelman  (OFPP  Administrator  of  Federal  Fiscal 
Policy)  identifies  the  two  most  common  complaints  among  contracting  officers  regarding 
the  use  of  past  performance  (ARNet,  1996:4).  First,  past  performance  is  an  imperfect 
predictor  of  the  future.  However,  both  the  public  and  private  sectors  have  found  this 
quality  approach  useful  in  achieving  best  value.  The  second  most  common  criticism  is 
that  past  performance  information  does  not  always  apply  to  the  proposal.  Thus  exists  the 
need  for  accurate  and  valid  relevancy  determinations.  A  more  defining  critique  considers 
relevance  with  respect  to  time.  Dr.  Kelman  suggests  recording  past  performance 
information  over  a  three-year-span  (ARNet,  1996:10).  In  addition,  past  performance 
evaluations  should  be  applied  to  commodities  (except  commercial  items)  and  some 
service  acquisitions,  but  not  for  procuring  major  systems  because  of  the  lack  of  timely 
relevance  to  similar,  forthcoming  source  selections  of  the  same  product  (Little,  1996:87). 

The  past  performance  evaluation  criteria  has  not  normally  been  a  major  factor  in 
the  final  award  decision.  An  investigation  of  each  evaluation  criteria  and  its  relationship 
with  the  final  award  decision  was  conducted  using  former  Air  Force  Logistics  Command 
source  selections.  Among  all  factors,  past  performance  had  the  lowest  correlation  with 
the  final  award  decision  (Noffsinger,  1991 :79-80).  It  may  have  been  that  all  offerors  had 
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decent  past  performance  ratings,  or  that  the  collection  methods  are  deficient  (Noffsinger, 
1991 :79-80).  The  Little  report  also  states  past  performance  is  “generally  not  a 
discriminator  in  the  selection  process”  and  estimates  a  5  to  15%  contribution  to  the  award 
decision  (1996:88). 

Overview  of  the  Source  Selection  Process 

Because  there  is  usually  more  than  one  method  to  deliver  certain  supplies  and 

services,  offerors  are  now  provided  with  objectives  and  they  return  a  proposal  to  fulfill 
these  requirements.  If  price  and  other  price  related  factors  were  the  only  evaluation 
factors  necessary  for  these  proposals,  then  the  contracting  method  should  be  “sealed 
bidding.”  However,  if  evaluation  of  these  proposals  should  be  made  on  the  basis  of 
evaluation  factors  besides  price  alone,  then  “contracting  by  negotiation”  applies.  “Source 
selection”  is  the  avenue  of  contracting  by  competitive  negotiation,  as  opposed  to  the  other 
avenue,  sole  source  acquisitions.  It  is  a  complex  and  thorough  manner  of  procuring 
systems  by  means  of  evaluating  proposals  in  terms  of  achieving  the  best  value  at  a  given 
price.  Source  selection  is  akin  to  a  major  personal  purchase;  one  does  not  always  choose 
a  brand  or  a  model  based  on  price  alone.  In  both  situations,  other  evaluation  factors  are 
invariably  represented  in  the  final  decision.  This  section  will  describe  the  source 
selection  process  in  order  to  understand  how  evaluation  factors  such  as  the  past 
performance  criterion  are  used. 

Statutory  guidance  for  conducting  source  selections  is  provided  in  the  Air  Force 
FAR  Supplement,  Appendix  AA — Formal  Source  Selection  for  Major  Acquisitions  and 
Appendix  BB — Source  Selection  Procedures  for  Other  Than  Major  Acquisitions.  In  both 
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types  of  acquisitions,  the  basic  overview  and  the  use  of  past  performance  is  much  the 


same. 

Players  involved  in  the  source  selection  process  include  the  Source  Selection 
Authority  (SSA),  Contracting  Officer  (CO),  Source  Selection  Advisory  Council  (SSAC), 
Source  Selection  Evaluation  Board  (SSEB),  Source  Selection  Evaluation  Team  (SSET), 
and  Performance  Risk  Assessment  Group  (PRAG).  The  designated  official  who  makes 
the  award  decision  is  the  Source  Selection  Authority.  This  person  is  advised  through  a 
group  of  senior  Government  officials,  or  the  Source  Selection  Advisory  Council. 
Together,  they  use  the  findings  and  proposal  evaluations  gathered  by  the  Source  Selection 
Evaluation  Board  for  recommendations  and  key  decisions.  In  a  streamlined  acquisition, 
the  combination  of  the  SSAC  and  the  SSEB  create  the  Source  Selection  Evaluation 
Team.  Figure  1  represents  a  typical  source  selection  organization. 


Figure  1.  A  Typical  Source  Selection  Organization 
(Adapted  from  AFFARS  Appendix  BB,  1996:BB-26) 
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A  Source  Selection  Plan  (SSP)  is  the  guiding  document  for  the  SSAC  and  SSEB, 
or  SSET.  Approved  by  the  SSA,  the  SSP  instructs  how  “the  proposals  will  be  evaluated 
and  analyzed,  and  how  the  source(s)  will  be  selected”  (AFFARS  Appendix  AA, 
1996:AA-3).  Among  its  minimum  contents  are  the  sections  entitled  “Evaluation 
Procedures”  and  “Evaluation  Criteria.”  Thus,  the  SSP  will  specify  the  “evaluation  and 
rating  methodology”  in  the  “Evaluation  Procedures,”  and  it  will  describe  the  cost 
criterion,  specific  criteria,  assessment  criteria,  proposal  risk,  performance  risk,  and 
general  considerations  under  the  “Evaluation  Criteria”  section  (AFFARS  Appendix  AA, 
1996:AA-12). 

As  to  the  evaluation  criteria  used  in  the  source  selection,  the  Federal  Acquisition 
Regulation  provides  instruction  when  making  awards  other  than  to  the  lowest-priced 
offeror.  Under  FAR  15.605,  Evaluation  Factors,  the  following  guidance  is  provided  for 
source  selection  methodology: 

While  the  lowest  price  or  lowest  total  cost  to  the  Government  is  properly  the 
deciding  factor  in  many  source  selections,  in  certain  acquisitions  the  Government 
may  select  the  source  whose  proposal  offers  the  greatest  value  to  the  Government 
in  terms  of  performance  and  other  factors.  (1994:  15.605(c)  ,  30,134) 

The  best  value  concept  is  embodied  in  the  established  guidelines  in  the  Federal 

Acquisition  Regulation,  which  allows  the  evaluations  to  be  based  on  “terms  of 

performance  and  other  factors”  (FAR  15.605(c)).  The  regulation  further  states,  “Past 

performance  shall  be  evaluated  in  all  competitively  negotiated  acquisitions  expected  to 

exceed  $100,000  not  later  than  January  1,  1999”  (FAR  15.605(b)(l)(ii)). 

Evaluation  criteria  are  the  basis  one-third  of  the  evaluation,  given  they  are  stated 

in  the  solicitation.  They  are  used  to  determine  the  factor  assessment.  The  three  types  of 
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evaluation  criteria  are  cost,  specific,  and  assessment  criteria.  The  cost  criteria  is 
mandatory  but  specific  criteria  are  related  to  the  specific  program  needs  and 
characteristics  and  therefore  used  when  they  are  applicable.  If  used,  specific  criteria  are  a 
subset  of  evaluation  criteria  and  are  “typically  divided  into  technical  and/or  management 
evaluation  areas.  Examples  of  specific  criteria  might  include  areas  such  as  technical, 
supportability,  manufacturing,  operational  utility,  design  approach,  readiness  and  support, 
test,  and  management”  (AFFARS  Appendix  AA,  1996:AA-1 1).  Assessment  criteria  are 
“used  by  evaluators  in  performing  the  technical  evaluation  by  relating  certain  aspects  of 
an  offeror’s  proposal  to  specific  evaluation  criteria”  (AFFARS  Appendix  AA,  1996:AA- 
2).  Examples  of  assessment  criteria  include  “soundness  of  approach,  completeness,  and 
compliance”  of  the  offeror’s  proposal  (AFFARS  Appendix  BB,  1996:BB-10).  As  stated, 
each  of  the  three  types  of  evaluation  criteria  are  used  to  determine  the  factor  assessment. 
In  addition  to  (and  on  equal  basis  of)  the  factor  assessment,  are  proposal  and  performance 
risks.  The  evaluation  matrix  in  Figure  2  demonstrates  that  the  factor  assessment, 
proposal  risk,  and  performance  risk  are  used  equally  to  evaluate  any  area  of  an  offeror’s 
proposal.  Proposal  risks  relate  the  risks  associated  with  the  offeror’s  proposed  approach. 
Performance  risks  are  defined  as  “the  assessment  of  an  offeror’s  present  and  past  work 
record  to  assess  confidence  in  the  offeror’s  ability  to  successfully  perform  as  proposed” 
(AFFARS  Appendix  AA,  1996:AA-2).  While  proposal  risks  identify  risks  in  the 
proposed  approach,  performance  risk  uses  an  offeror’s  track  record  to  assess  a  confidence 
level  in  the  offeror’s  ability  to  perform  as  promised. 

The  Air  Force  uses  a  color  rating  system  for  specific  and  assessment  criteria.  The 
colors  are:  Blue,  or  Exceptional  (“Exceeds  specified  performance  or  capability  in  a 
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beneficial  way  to  the  Air  Force”);  Green,  or  Acceptable  (“Meets  evaluation  standards”); 
Yellow,  or  Marginal  (“Fails  to  meet  evaluation  standards”);  and  Red,  or  Unacceptable 
(“Fails  to  meet  a  minimum  requirement”  and  the  deficiency  is  not  correctable)  (AFFARS 
Appendix  AA,  1996:AA-15).  Proposal  and  performance  risk  are  not  rated  in  a  color 
scheme,  in  contrast  to  the  color  system  for  specific  and  assessment  criteria.  These  two 
risk  types  are  rated  amongst  a  high,  moderate,  and  low  spectrum.  Figure  2  demonstrates 
this  differentiation. 


Area  X 

Factor  N  (e.  g.,  Technical) 


Factor 

Green 

Assessment 

(Meets  Standards) 

Proposal 

Moderate 

Risk 

(Potential  Difficulties) 

Past 

Low 

Performance 

(Little  doubt  that  the  offeror 

can  perform  the  effort) 

Figure  2.  Example  of  an  Evaluation  Matrix 
(Adapted  from  AFFARS  Appendix  AA,  1996:AA-31) 


Proposal  risk  relates  the  proposed  approach  to  the  assessment  and  specific  criteria. 
A  high  proposal  risk  rating  implies  the  proposal  is  prone  to  cause  significant  schedule 
delay,  increased  costs,  or  poor  performance  even  with  close  contractor  monitoring.  A 
moderate  risk  rating  signifies  potential  difficulties  in  schedule,  cost,  or  performance. 
Special  contractor  monitoring  by  the  Government  may  overcome  these  difficulties.  A 
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low  rating  represents  little  potential  for  these  difficulties.  It  implies  normal  contractor 
monitoring  will  probably  overcome  difficulties. 

Performance  risk  relates  cost  and  specific  criteria  and  these  ratings  are  based  on 
the  offeror’s  past  and  present  performance  record.  A  high  performance  risk  rating 
implies  significant  doubt  that  the  offeror  can  perform  the  proposed  effort.  A  moderate 
performance  risk  rating  implies  the  some  doubt  exists  as  to  the  performance  of  the 
proposed  effort.  A  low  rating  projects  little  doubt  that  the  offeror  can  perform  the  effort. 
An  additional  category,  not  applicable,  implies  no  significant  performance  record  is 
identifiable. 

In  regards  to  past  performance,  AFFARS  Appendices  AA  and  BB  recognize  that 
performance  risk  evaluation  is  vital  to  the  award  decision.  The  policy  for  past 
performance  dictates  that  “the  assessment  of  past  performance  (1/3)  is  of  equal 
importance  to  either  factor  assessment  (1/3)  or  proposal  risk  (1/3)”  (1996:AA-4).  The 
policy  further  emphasizes  “the  Air  Force’s  commitment  to  award  only  to  contractors  who 
will  carry  through  with  what  they  promise  in  their  proposals”  (1996:AA-4).  Rationale  for 
using  past  performance  is  stated  as  to  “prevent  awards  to  habitually  poor  performers,  and 
reduce  the  incidence  of  associated  schedule  slips  and  increased  costs”  (1996:AA-4). 

An  evaluation  matrix  graphically  displays  the  factor  assessment,  proposal  risk, 
and  past  performance.  The  matrix  shows  that  each  of  these  aspects  are  given  the  same 
weight.  In  Figure  2,  the  evaluation  matrix  indicates  that  for  the  technical  area,  the 
assessment  meets  standards,  the  proposal  bears  moderate  potential  for  difficulties,  and 
small  doubt  exists  that  the  offeror  can  perform  the  effort. 
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Once  the  Best  And  Final  Offers  have  been  submitted  and  the  evaluations  have 


been  completed,  the  SSEB  chairperson  briefs  the  SSAC  with  a  written  presentation,  or 
the  Proposal  Analysis  Report  (PAR)  (AFFARS  Appendix  AA,  1996:AA-20).  The  PAR 
documents  the  results  of  the  technical  evaluation,  risk  assessments,  cost  analysis,  contract 
issues,  SSAC  analysis,  findings,  and  rationale  (AFFARS  Appendix  AA,  1996:AA-2).  It 
contains  evaluation  elements  including  the  proposal  ratings,  proposal  risk  assessments, 
performance  risk  assessments,  and  comments  of  proposal  strengths  and  weaknesses.  The 
SSEB  chairperson  also  provides  an  oral  presentation  which  should  include  assessments  of 
the  technical  evaluation,  cost  analysis,  general  considerations,  performance  risk,  and  the 
SSAC’s  overall  independent  comparative  analysis  of  the  proposals  (AFFARS  Appendix 
AA,  1996:AA-21). 

A  case  study  highlighted  the  power  of  the  SSA’s  decision,  particularly  when  the 
SSA  is  presented  with  erroneous  information  and  decides  against  a  contractor  with  an 
inaccurate  performance  record.  Though  the  contractor’s  past  performance  record  was 
incorrect,  the  case  summarized  the  discretion  and  finality  of  the  SSA’s  decision  (Witte, 
1989:38).  The  resolution  of  the  case  was  that  the  SSA  made  a  rational  decision  at  the 
time  of  the  source  selection  and  his  judgment  was  upheld  based  on  these  facts  (1989:38). 

After  the  SSA  has  made  the  award  decision,  debriefings  are  held  with  each 
offeror.  Disclosing  of  the  past  performance  information  to  the  offeror  ensures  fairness, 
provides  the  offeror  steps  towards  continuous  improvement,  and  fosters  better  supplier 
relationships.  It  is  not  necessary  to  disclose  to  the  offeror  the  exact  source  from  where 
the  past  performance  information  was  collected. 
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Performance  Risk  Assessment 


A  source  selection  is  conducted  to  select  the  offeror  with  the  best  value  proposal. 
Another  function  is  to  make  the  selection  with  confidence  that  the  offeror  has  the  ability 
and  willingness  to  deliver  what  has  been  promised  in  the  proposal.  An  evaluation  of  past 
performance  involves  three  aspects:  a)  objective  facts  of  the  contractor’s  history;  b) 
subjective  judgments  about  how  well  the  contractor  satisfied  the  customer;  and  c) 
subjective  judgments  about  the  information’s  relevance  (Edwards,  1995:25).  It  indicates 
a  confidence  that  the  offeror  will  perform  as  promised  in  the  proposal.  This  confidence  is 
conventionally  known  as  the  performance  risk  assessment. 

Regulatory  guidelines  for  assessing  performance  risk  comes  from  the  Air  Force 
FAR  Supplement.  All  assessments  will  recognize  the  “number  and  severity  of  problems, 
the  effectiveness  of  corrective  actions  taken,  and  the  overall  work  record”  to  measure  how 
well  a  contractor  will  overcome  difficulties  upon  close  Government  monitoring 
(1996:AA-16).  Also  considered  is  the  contractor’s  demonstrated  ability  to  identify  and 
correct  program  risks.  Performance  risk  will  be  tailored  to  cost  and  specific  criteria,  and 
will  be  discussed  separately  in  the  evaluation  briefings. 

Past  performance  information  only  indicates  the  contractor’s  demonstrated 
actions.  Arthur  D.  Little,  Inc.  categorizes  past  performance  information  as  the  record  of 
specification  conformance,  cost  maintenance  and  forecasting  on  cost  reimbursable 
contracts,  delivery  schedule  compliance,  history  of  behavior,  and  established  concern  for 
the  customer  (1996:12).  Dr.  Kelman,  OFPP  Administrator  of  Federal  Fiscal  Policy, 
suggests  that  for  those  contractors  large  enough  to  contain  divisions  or  profit  centers, 
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consideration  of  past  performance  information  should  reflect  only  the  particular  division 
or  profit  center  who  will  perform  the  work  on  the  contract  (ARNet,  1996:16).  General 
categories  of  demonstrated  past  performance  have  been  outlined  by  Dr.  Kelman.  This  list 
includes  quality,  timeliness,  cost  control,  cooperation  with  the  contracting  officer, 
customer  satisfaction,  and  key  personnel’s  management  of  the  contract  (ARNet, 

1996:14).  Dr.  Kelman  also  outlines  the  categories  of  contractor  performance  information 
that  are  required  for  records  on  every  contract  exceeding  $100,000.  The  categorization  is 
described  in  Figure  3. 


Quality 

Conformance  to  specifications  and 
standards  of  good  workmanship 

Cost  Control 

Containment  and  forecasting 
of  costs 

Timeliness 

Adherence  to  contract  schedules, 
including  administrative  aspects 

Business  Relations 

History  of  reasonable  behavior 
and  overall  concern  for  the  customer 

Customer  Satisfaction 

Service  to  the  end  user  of  the 
product  or  service 

Key  Personnel 

How  long  they  stayed  on  the  contract; 
how  well  they  managed  their  portion 

Figure  3.  Categories  for  Recording  Past  Performance  Information 
(Adapted  from  ARNet,  1996:9  and  35) 
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The  Contractor  Performance  Assessment  Report,  which  systematically  assesses 
contractor  performance  for  acquisitions  greater  than  $5  million,  also  categorizes  past 
performance  information.  These  categories  of  performance  assessments  include 
management  responsiveness,  schedule  control,  technical  performance  of  product,  product 
assurance  (both  quality  system  and  manufacturing  management),  program/data 
management,  procurement/subcontract  management,  logistics  support/sustainment, 
engineering  (both  system  and  software  engineering),  and  unique  technical 
performance/other  areas  (AFMCI  64-107,  1996: Attachment  1). 

Past  performance  information  is  collected  for  two  different  uses.  Information  is 
collected  by  procurement  agencies  during  a  local  source  selection,  and  it  is  also  collected 
for  future  uses  (Little,  1996:64).  Figure  4  demonstrates  how  past  performance 
information  is  collected  for  these  two  different  uses. 
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Collection  of  Past 
Performance  Information 
For  Future  Use 


Collection  and  Use  of  Past 
Performance  Information  During 
Contractor  Selection 


Figure  4.  Two  Different  Treatments  of  Past  Performance  Information 
(Adapted  from  Little,  1996:64) 

In  either  circumstance,  the  past  performance  information  shall  be  treated  as 
sensitive  source  selection  information  and  is  not  releasable  under  the  Freedom  of 
Information  Act. 

Past  performance  information  can  be  used  in  source  selections  and  for 
responsibility  determinations.  However,  a  responsibility  determination  cannot  be  used  to 
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assess  performance  risk.  A  responsibility  determination  implies  that  the  contractor  has  a 
“satisfactory  performance  record”  (FAR  9. 104- 1(c)).  It  ensures  that  the  offeror  has  the 
capability  to  perform  and  screens  out  the  non-responsible  offerors  who  do  not  have  it. 
Using  performance  risk  assessments  during  a  source  selection  provides  a  competitive 
method  to  compare  each  offeror’s  track  record  with  other  responsible  offerors  in 
determining  which  one  will  provide  the  best  value. 

The  Performance  Risk  Assessment  Group  (PRAG)  was  established  as  a 
Government’s  response  to  commercialize  the  practices  involved  in  source  selections.  The 
U.  S.  Army  uses  a  PRAG  which  is  very  similar  to  the  Air  Force  version  (Dept  of  Army 
“Past,”  1996).  Usually  consisting  of  one  chairperson  (Lieutenant  Colonel  or  GS 
equivalent)  and  two  assistants,  the  PRAG  is  a  temporary  group  assigned  with  the 
responsibility  of  performing  the  source  selection’s  performance  risk  assessment.  The 
overall  process  involves  subjective  judgment  and  imprecise  methodology.  PRAG 
members  should  have  cost,  procurement,  and/or  technical  backgrounds.  The  PRAG 
separately  briefs  the  SSAC  or  SSET  Chairperson.  All  PRAG’s  usually  follow  the 
following  same  steps:  planning,  obtaining  data,  supporting  the  competitive  range 
briefing,  compiling  and  evaluating  information,  assessing  risk,  identifying  concerns  to  the 
contracting  officer  for  discussions,  and  briefing  the  SSA  (Thurston,  1989:4-2).  If  past 
performance  information  from  contracts  performed  on  federal,  state,  or  local  contracts  are 
not  available,  then  PRAG  members  should  use  commercial  contracts  if  practical. 
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Sources  of  Past  Performance  Information 


During  the  source  selection,  past  performance  information  can  be  obtained 
through  a  myriad  of  sources.  Three  types  of  information  sources  that  are  used  throughout 
the  federal  government  are:  appraisal,  tracking,  and  certifying  systems  (Little,  1996:14). 
Performance  appraisal  systems  contain  contractor  evaluations;  tracking  systems  contain 
quality  and  delivery  data;  certifying  systems  establish  specific  criteria  and  maintain  a  list 
of  contractors  who  have  demonstrated  the  criteria  in  previous  work.  Brislawn  and  Dowd 
cite  that  a  single  source  of  past  performance  information  is  not  an  adequate  indicator  of 
the  contractor’s  past  performance  history  (1996:18).  Instead,  they  list  multiple  sources  of 
past  performance  information  to  include:  contract  completion  evaluations,  performance 
ratings  from  CPARS  or  DPRO’s,  contractor  self-assessments,  user  and  buyer  evaluations, 
and  performance  qualifications  and/or  certifications  (1996:18).  An  outline  of  available 
information  sources  includes  the  “offeror’s  proposal,  Contractor  Performance  Evaluation 
Program  (CPEP)  reports,  Contractor  Performance  Assessment  Reports  (CPARS),  the 
Acquisition  Management  Information  System  (AMIS),  and  the  Defense  Logistics  Agency 
(DLA)  contractor  profile  system”  (Biddy,  1995:6).  The  PRAG  members  also  use 
interviews  and  questionnaires  to  extract  past  performance  information.  Among  the 
documents  for  past  performance  risk  determinations,  Noffsinger  noted  the  use  of  “1)  the 
findings  of  the  PRAG;  2)  the  Contractor  Responsibility  Review  Program;  3)  CPARS;  4) 
questionnaires;  5)  interviews;  6)  pre-award  surveys;  and  7)  reviews  of  contractor 
performance  on  similar  contracts”  (1991 :63).  Financial  reports  may  also  provide 


26 


information  of  the  offeror’s  stability/instability  or  strengths/ weaknesses  (Edwards, 
1995:37). 

The  major  steps  of  obtaining  past  performance  information  are  geared  towards 
collecting  information  and  then  validating  it.  Little  cites  the  following  five  most  common 
steps  of  obtaining  past  performance  information  through  existing  systems:  1)  evaluating 
and  recording  performance;  2)  providing  contractors  an  opportunity  for  rebuttal;  3) 
receiving  contractor  rebuttals;  4)  reviewing  the  rebuttal  and  resolving  the  evaluation;  and 
5)  filing  all  information  as  “Source  Selection  Information”  for  use  within  a  three-year 
period  (1996:9). 

The  Contract  Performance  Assessment  Reporting  System  (CPARS)  was  also 
developed  as  an  manual  contractor  performance  tracking  system.  Used  by  Air  Force 
Materiel  Command,  it  is  a  database  for  “systematically  assessing  contractor  performance 
on  current  system  acquisitions  greater  than  $5  million”  (AFMC  Instruction  64-107, 
1996:1).  The  purpose  of  the  CPARS  is  to  facilitate  an  on-going  database  kept  at  Air 
Force  Product  Centers  by  CPAR  focal  points  for  responsibility  determinations,  formal, 
and  informal  source  selections.  CPARS  is  updated  every  twelve  months.  A  contractor’s 
record  in  the  CPARS  will  include  objective  facts  such  as  those  generated  by  cost 
performance  reports,  schedule  compliance,  customer  complaints,  quality  reviews, 
production  management  reviews,  functional  performance  evaluations,  and  subcontract 
management.  The  entry  will  also  include  a  subjective  explanation  as  to  the  cause  of  the 
negative  remarks.  The  Government  and  the  contractor’s  perspectives  are  presented  on  the 
same  form,  but  the  evaluations  are  not  subject  to  the  “Disputes”  clause  nor  appeals. 
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Dr.  Steven  J.  Kelman  states  that  contracting  agencies  must  establish  a  review 
procedure  for  contractors’  comments  that  is  at  least  one  level  above  the  contracting 
officer  and  resolution  is  up  to  the  contracting  agency  (1996:9). 

Another  source  of  past  performance  information  is  the  Contractor  Performance 
Evaluation  Program  (CPEP).  This  is  a  contract  administration  program  which  stores 
contractor  performance  information  during  and  after  the  completion  of  contracts. 
Contractors  are  given  an  assessment  twice  a  year.  CPEP  applies  to  contracts  between  $2 
million  to  $5  million  (Biddy,  1995:12). 

PRAG  members  must  make  the  effort  to  find  contracts  not  listed  in  the  offeror’s 
proposal  as  additional  sources  of  performance  history.  Contracts  previously  held  by  the 
contractor  can  be  found  through  the  Acquisition  Management  Information  System 
(AMIS).  This  automated  database  contains  contract  information  with  respect  to  number, 
title,  and  dollar  amount  for  all  corporations  on  contract  with  AFMC  (Biddy,  1995:12). 
The  Federal  Procurement  Data  System  (FPDS),  managed  by  the  General  Services 
Administration,  serves  as  another  source  of  each  offeror’s  federal  contract  history.  The 
FPDS  keeps  award  information  reported  on  a  Standard  Form  279  or  DD  350  for  every 
contract  in  excess  of  $25,000  (Edwards,  1995:37  and  FAR  4.6).  Legal  databases  such  as 
LEXIS  or  WESTLAW  may  also  serve  as  an  information  source  for  contractors  currently 
involved  in  litigation. 

The  Contractor  Profile  System  (CPS)  at  the  Defense  Logistics  Agency  (DLA)  is 
another  past  performance  system  currently  available,  but  its  expected  replacement  is  the 
Defense  Contract  Management  Command  Service’s  Contractor  Information  Service 
(CIS).  The  current  CPS  is  a  broad  database  useful  for  sharing  information  within  the 
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services  and  contains  “data  on  production,  quality,  contract  administration,  financial 
services,  engineering,  property,  procurement,  and  legal”  (Biddy,  1995:12). 

The  DLA  also  has  systems  which  serves  a  notice  to  Contracting  Officers  that  a 
condition  exists  needing  consideration  prior  to  award.  The  Contractor  Alert  List  (CAL) 
is  a  list  of  contractors  who  were  placed  in  the  Contractor  Improvement  Program  (CIP)  or 
recommended  by  DCMC  for  a  pre-award  survey  (Biddy,  1995: 12).  The  DLA  Quality 
Alert  List  identifies  contractors  who  have  not  corrected  a  problem  within  their  inspection 
system  or  quality  system  (Biddy,  1995:13).  Contractors  are  on  this  list  until  the  identified 
problem  has  been  resolved. 

The  primary  means  of  collecting  information  for  the  lower  dollar  acquisitions  is 
through  interviews  and  questionnaires.  However,  these  methods  are  time-consuming 
methods  for  gaining  past  performance  information  from  each  offeror.  Questionnaires 
generate  factual  information  from  the  offeror  related  to  past  contracts  (e.  g.  number, 
dollar  amount,  points  of  contact)  and  subjective  information  related  to  their  experience  (e. 
g.  a  unique  technical  aspect  related  to  the  proposal,  or  reasons  for  a  schedule  slip).  The 
follow-on  interviews  given  to  various  respective,  knowledgeable  government  personnel 
are  more  extensive  and  practical  to  the  final  risk  assessment  (Biddy,  1995:36-37). 
Questions  regarding  the  technical  aspects  normally  invoke  the  respondent’s  subjective 
assessment  of  the  contractor’s  methodology,  test  program,  and  logistics  support 
capability.  Reliability,  maintainability,  and  producibility  are  also  usually  probed.  Each 
offeror  is  also  assessed  in  areas  such  as  management  of  contract  requirements, 
subcontractors,  and  evaluation  change  proposals,  support  to  Government  personnel,  and 


29 


adherence  to  small  or  disadvantaged  business  goals.  Questions  regarding  schedule  and 
cost  goals  and  the  reasons  for  not  meeting  these  goals  are  also  often  asked. 

Certain  strengths  and  weaknesses  exist  in  the  attempts  to  extract  performance 
information.  The  most  troublesome  area  is  usually  questionnaires  and  interviews.  The 
problems  encountered  involve  the  amount  of  time  to  locate  the  right  person  to 
survey/interview,  respondent  reluctance  to  phone  interviews,  inability  to  ascertain 
respondent  objectivity,  and  ineffectively  communicating  the  questions  (Thurston,  1989:4- 
7).  Moreover,  the  most  difficult  challenge  is  finding  sufficient  reliable  information  to 
make  an  intelligent  assessment  of  an  offeror’s  performance  risk  (Edwards,  1995:60). 

Relevancy  Determinations 

Once  the  fact-finding  is  complete,  it  is  important  for  the  PRAG  to  determine  if  the 
information  is  relevant.  Relevancy  is  defined  as  “How  closely  the  skill  demonstrated  in 
the  prior  contract,  e.  g.,  subcontract  management,  matches  the  degree  to  which  that  skill 
will  be  utilized  on  the  new  contract”  (AFMCPAM  64-113,  1995:6).  Subjective  ratings  of 
Relevant,  Somewhat  Relevant,  and  Not  Relevant  are  used  to  underscore  the  relevancy  of 
the  past  performance  information  to  the  Government’s  requirement  at  hand  (Biddy, 

1 995: 1 1).  The  contractor’s  relevancy  rating  should  be  reflected  in  the  overall 
performance  risk  assessment. 

Problem  Restatement 

How  have  AFMC  acquisition  professionals  been  implementing  past  performance 
in  their  source  selections  between  $1,000,000  and  $5,000,000?  Moreover,  what 
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suggestions  can  be  made  to  improve  the  process?  In  order  to  answer,  the  following 
questions  need  investigation:  What  methods  of  collecting,  storing,  and  protecting  past 
performance  information  are  currently  being  practiced?  What  kinds  of  past  performance 
information  have  been  used?  How  do  they  determine  the  information’s  relevancy? 
Essentially,  what  are  the  similarities  and  differences  between  past  performance 
information  systems  for  the  lower-dollar  acquisitions?  With  the  answer  to  these 
questions,  a  foundation  can  be  made  for  developing  a  policy  for  the  implementation  of 
past  performance  practices  in  the  $1  million  to  $5  million  acquisition  range. 
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III.  Research  Methodology 


Introduction 

As  discussed  in  Chapter  2,  the  past  performance  criterion  has  demonstrated  itself 
as  an  effective  indicator  of  best  expected  value  for  reasons  such  as  evaluating 
performance  risk,  improving  contractor  performance,  and  providing  more  effective 
management  of  contractor  performance.  Chapter  2  also  discussed  how  the  Air  Force 
established  the  Contractor  Performance  Assessment  Reporting  System  (CPARS)  as  a 
centralized  database  for  storing  past  performance  information  on  acquisitions  above  $5 
million.  However,  there  is  no  guidance  for  collecting  past  performance  information  for 
acquisitions  below  this  amount.  Therefore,  it  is  hypothesized  that  some  work  centers  are 
collecting  the  information  for  “in-house”  purposes.  Literature  research  indicates  that  the 
value  of  past  performance  information  decreases  as  a  past  performance  system  is 
standardized  and  therefore  the  system  should  be  kept  at  the  agency  level.  Furthermore, 
past  performance  information  should  reflect  each  acquisition’s  business  area.  This 
research  effort  will  investigate  the  similarities  and  differences  among  procedures  for 
implementing  past  performance  evaluation  requirements  for  the  lower  dollar  acquisitions. 

Objectives 

Chapter  1  introduced  the  following  investigative  questions:  How  are  the 
Contracting  Officers  storing  their  past  performance  information?;  How  are  the 
Contracting  Officers  protecting  the  past  performance  information?;  How  are  the 
Contracting  Officers  making  their  relevancy  determinations?;  What  sources  of  collecting 
past  performance  information  are  being  used?;  What  questions  are  being  asked  when 
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collecting  the  information?;  and  What  similarities  and  differences  exist  between  past 
performance  information  systems?  The  answers  to  these  questions  will  reveal  the  past 
performance  evaluation  procedures  currently  used  for  the  lower  dollar  acquisitions  by 
AFMC  work  centers.  It  is  hypothesized  that  there  is  no  standard  information  system  to 
handle  past  performance  information.  An  analysis  will  investigate  the  similarities  and 
differences  between  the  past  performance  information  systems.  From  this  analysis, 
suggestions  will  be  made  for  future,  lower  dollar  AFMC  acquisitions. 

These  investigative  questions  lead  to  the  following  six  objectives  for 
understanding  the  implementation  of  past  performance  information  and  evaluations  in 
acquisitions  under  $5  million: 

Objective  #1:  Storing  the  Information; 

Objective  #2:  Protecting  the  Information; 

Objective  #3:  Relevancy  Determinations; 

Objective  #4:  Sources  of  Past  Performance  Information; 

Objective  #5:  Past  Performance  Information  Types;  and 

Objective  #6:  Similarities  and  Differences  Between  Past  Performance  Information 
Systems. 

Objective  #1 :  Storing  the  Information 

Since  the  collection  of  past  performance  information  is  not  regulated  with  a 
systematic,  centralized  database  such  as  the  Contractor  Performance  Assessment 
Reporting  System  (CPARS),  the  work  centers  must  store  their  past  performance 
information  in  a  de-centralized  database.  As  stated  earlier,  there  is  no  regulatory 
guidance  to  follow  for  the  collection  of  past  performance  information  below  $5  million. 


Through  interviews,  this  objective  will  describe  the  various  commonalities  and 
differences  in  the  methods  that  the  contracting  professionals  have  been  using  to  store  the 
information.  It  will  test  the  hypothesis  that  some  work  centers  have  constructed  their 
own  “in-house”  database.  It  will  also  describe  how  often  information  is  input,  updated, 
or  removed.  It  is  further  necessary  to  probe  reasons  against  the  decision  to  maintain  an 
“in-house”  database. 

Objective  #2:  Protecting  the  Information 

The  CPARS  regulation  mandates  that  “All  Contractor  Performance  Assessment 
Report  forms,  attachments,  and  working  papers  must  be  marked  ‘FOR  OFFICIAL  USE 
ONLY/SOURCE  SELECTION  INFORMATION”  (AFMCI  64-107,  1996:6).  Thus,  the 
CPARS  regulation  specifies  that  past  performance  information  must  be  kept  confidential, 
but  what  are  the  specific  protection  methods  for  acquisitions  under  $5  million?  The 
questions,  “Who  has  access  to  input  information?”  and  “Who  has  access  to  retrieve 
information?”  are  necessary  to  identify  specific  protection  methods.  The  question,  “Do 
contractors  have  the  opportunity  to  review  their  files?,”  was  asked  to  identify  if  a 
difference  exists  between  those  work  centers  who  lean  towards  protection  and  those  who 
invite  contractors  to  review  their  performance  information.  The  purpose  of  “What  is  your 
policy  towards  another  work  center  asking  you  for  information  on  a  particular 
contractor?”  is  to  understand  the  nature  of  sharing  past  performance  information  among 
AFMC  work  centers.  Interviews  were  used  to  glean  protection  methods  and  issues. 
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Objective  #3:  Relevancy  Determinations 


A  relevancy  determination  is  a  measure  taken  to  match  the  degree  to  which  an 
offeror’s  past  performance  matches  the  skill  required  under  the  new  contract.  It  is  not 
assumed  that  the  function  of  performing  relevancy  determinations  remains  the  same  as 
that  for  the  higher-dollar  acquisitions.  Different  criteria  may  apply.  Generally, 
acquisitions  above  $5  million  involve  more  unique  requirements  rather  than  commercial 
products  or  services.  It  is  the  interest  of  this  objective  to  establish  which  criteria  are 
useful  in  performing  relevancy  determinations  for  the  lower  dollar  acquisitions.  The 
buyers  and  contracting  officers  face  certain  challenges  in  regards  to  using  relevant 
information  during  evaluations.  Therefore,  it  is  of  interest  to  present  these  challenges  to 
understand  the  decision  to  implement  a  database  or  to  gather  past  performance 
information  through  other  means.  Interviews  were  used  to  describe  commonalities  and 
differences. 

Objective  #4:  Sources  of  Past  Performance  Information 

Before  establishing  what  specific  information  is  used  in  performance  evaluations, 
the  sources  first  need  to  be  identified.  It  is  hypothesized  each  information  source  is 
useful  to  some  but  not  all  work  centers  in  collecting  past  performance  information.  The 
hypothesized  sources  were  derived  from  a  combination  of  literary  sources  documented  in 
Chapter  2.  The  sources  actually  used  by  AFMC  work  centers  were  obtained  by  using 
interviews.  The  interview  was  conducted  in  accordance  with  Appendix  A. 
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Objective  #5:  Past  Performance  Information  Types 


This  objective  explores  the  commonalities  and  differences  in  past  performance 
information  types  among  the  various  work  centers  and  answers  the  question:  What  kinds 
of  past  performance  information  are  used  in  evaluations?  The  work  centers  have  devised 
their  own  methods  of  collecting  and  using  past  performance  information  since  no 
guidance  exists  for  acquisitions  less  than  $5  million.  It  is  hypothesized  that  the  work 
centers  use  the  same  past  performance  information  types  as  categorized  in  the  CPAR 
format  combined  with  Dr.  Kelman’s  (OFPP  Administer  of  Federal  Procurement  Policy) 
list  of  information  types. 

These  five  objectives  present  an  outline  of  how  AFMC  work  centers  are  handling 
past  performance  in  the  $1  million  to  $5  million  acquisition  range.  This  outline  covers 
the  storage  and  protection  methods  along  with  a  review  of  useful  information  types, 
sources,  and  relevancy  criteria.  The  first  five  objectives  provide  a  baseline  for  defining, 
understanding,  and  referencing  how  AFMC,  in  general,  has  responded  to  past 
performance  practices.  This  leads  to  Objective  #6,  which  will  answer  the  question:  Are 
some  work  centers  finding  different  means  to  handle  past  performance  information? 

Objective  #6:  Similarities  and  Differences  Between  Past  Performance  Information 
Systems 

It  is  hypothesized  that  several,  but  not  all,  work  centers  have  constructed  an  in- 
house  database  to  collect  and  maintain  their  own  repository  of  past  performance 
information.  The  variance  surrounding  this  decision  on  storing  the  information  may 
imply  that  subsequent  decisions  (i.  e.,  protecting  the  information,  relevancy,  information 
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sources  and  types)  differ  as  well.  The  following  independent  tests  were  performed  to 
delineate  the  similarities  and  differences  among  the  various  information  systems  used  by 
AFMC  work  centers: 

Test  A:  Do  the  information  systems  offer  the  contractor  an  opportunity  to  review 
the  information? 

Test  B:  Do  the  information  systems  provide  the  same  level  of  authorization  to 
input  or  retrieve  information? 

Test  C:  Do  the  information  systems  use  the  same  relevancy  criteria? 

Test  D:  Do  all  work  centers  use  the  same  sources  of  past  performance 
information  regardless  if  they  use  an  in-house  database  or  not? 

Test  E:  Do  the  information  systems  use  the  same  past  performance  information 
types? 

Indexing  the  Results  for  Analysis  of  Similarities  and  Differences  Between  Past 
Performance  Information  Systems 

Tests  A  through  E  in  Objective  #6  concern  the  similarities  and  differences  past 
performance  information  systems.  In  order  to  analyze  these  tests,  it  is  necessary  to  sub¬ 
categorize  each  of  the  tabulated  answers  (from  the  interview  questions  for  the  first  five 
objectives)  to  the  five  tests  under  each  respective  information  system.  An  analysis  of 
these  similarities  and  differences  was  performed  by  using  independent  tests  with  an 
indexing  system. 

The  Non-numerical  Unstructured  Data  Indexing  Searching  and  Theorizing  (QSR 
NUD*IST)  software  is  useful  in  “handling  non-numerical  and  unstructured  data  in 
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qualitative  analysis”  (QSR  NUD*IST,  1996:2).  The  software  was  used  to  create  an 
indexing  system  for  analyzing  the  results  from  Objectives  #l-#5.  The  index  is  similar  to 
a  database  in  that  it  links  categories  and  sub-categories  such  as  past  performance 
information  types  with  each  respective  source.  This  software  was  used  to  test  similarities 
and  differences  among  the  various  information  systems  used  by  AFMC  work  centers. 


The  Sample 

Eleven  contracting  personnel  among  AFMC  product  centers  and  air  logistics 
centers  were  interviewed.  Points  of  contact  were  provided  by  HQ  AFMC  to  help  identify 
respondents  who  would  be  the  most  knowledgeable  about  past  performance  and  its 
implementation  within  their  organization.  Certain  points  of  contact  could  not  be  located 
(e.  g.,  relocation  to  a  new  position).  In  this  event,  a  Contractor  Performance  Assessment 
Reporting  System  (CPARS)  focal  point  was  sought.  If  that  was  not  feasible,  more  than 
one  interview  was  conducted  for  that  work  center.  All  interviewees  are  knowledgeable 
about  how  past  performance  policies  have  been  implemented  within  their  work  center. 
The  interviewees  are  Government  civilian  employees  with  five  to  twenty  years  of 
contracting  experience.  About  half  of  these  people  serve  as  the  CPARS  focal  point.  The 
Air  Force  Material  Command  is  represented  with  two  product  centers’  and  five  air 
logistic  centers’  participation  in  this  research  effort.  Two  product  centers  did  not 
participate  because  they  do  not  perform  acquisitions  within  the  $1  million  to  $5  million 
range.  These  centers  were  identified  by  HQ  AFMC.  The  actual  work  centers  who 
participated  in  this  research  effort  are  anonymous.  No  work  center  will  be  singled  out  in 
regards  to  the  policies  or  decisions  they  have  implemented.  This  aspect  was  explained  to 


each  of  the  interviewees  in  the  course  of  providing  an  open  dialogue  rather  than  an 
inquisition. 

Interviews 

For  each  of  the  first  five  objectives,  the  interview  technique  is  most  appropriate 
and  feasible.  Interviews  are  appropriate  for  obtaining  data  which  represents  actual 
policies  and  the  respondents’  beliefs  and  feelings  about  them  (Silverman,  1993:91,92). 
They  are  also  appropriate  for  “specific  questions  related  to  actual  rather  than  hypothetical 
situations”  (Silverman,  1993:92).  The  purpose  of  this  research  effort  is  to  identify  how 
AFMC  work  centers  have  implemented  past  performance  information  systems  within 
their  work  center.  It  is  also  necessary  to  discern  what  information  sources  and  types  are 
considered  “useful”  for  the  $1  million  to  $5  million  acquisition  range.  Therefore, 
interviews  are  appropriate. 

Interviews  are  used  as  a  qualitative  research  method  for  two  different  positions: 
Positivism  and  Interactionism  (Silverman,  1993:90,91).  Through  positivism,  an 
interview  provides  facts  about  behavior  and  attitudes  by  tabulating  multiple-choice 
answers  to  standardized  questions  (Silverman,  1993:91).  A  few  of  the  questions  listed  in 
Appendix  A  obtain  data  this  way  (e.  g..  Is  the  information  stored  electronically  or 
manually?;  Does  a  contractor  have  the  opportunity  to  review  the  file?) 

Interactionism  is  an  interview  position  which  serves  to  obtain  the  respondents’ 
insights  by  using  unstructured,  open-ended  questions  (Silverman,  1993:91).  Most  of  the 
interview  questions  listed  in  Attachment  A  serve  this  purpose.  Open-ended  questions 
were  used  in  providing  unique,  in-depth  data  from  the  respondents  (e.  g.,  What  challenges 
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do  you  face?).  However,  using  these  types  of  questions  limits  reliability  and  validity 
unless  the  researcher  introduces  passivity  while  eliciting  the  response  (Silverman, 
1993:96).  Therefore,  each  question  was  read  as  it  was  worded  in  Appendix  A.  To  a 
small  degree,  some  questions  needed  further  background  to  accurately  pose  the  same 
query  to  each  respondent. 

Because  interviews  involve  an  interactive  flexibility  with  the  respondents,  overall 
reliability  and  validity  may  be  subject  to  challenge.  A  standardized  protocol  was  used  to 
minimize  this  flexibility  (Silverman,  1993:92).  The  respondents  were  each  given  the 
same  interview  (See  Appendix  A).  Questions  were  read  in  the  sequence  listed  in  this 
attachment.  When  feasible,  interviews  were  tape-recorded  at  the  respondents’  office  with 
their  permission.  The  purpose  of  the  tape  recorder  is  to  offer  more  reliable  transcription 
of  the  data  for  coding  and  analyses.  The  respondents  were  each  given  the  opportunity  to 
turn  off  the  tape  recorder  at  any  time  they  chose  or  to  turn  off  the  recorder  completely. 
Most  of  the  interviews  were  conducted  over  the  telephone  as  time  and  funding  limited  the 
capability  to  visit  each  respondent  at  their  location.  The  details  of  the  interview  were 
confirmed  with  the  respondents  and  then  the  transcripts  were  coded  and  analyzed. 

Interview  Questions 

The  questions  found  in  the  interview  format,  Appendix  A,  were  selected  to 
compare  the  data’s  relationship  with  AFMC  Instruction  64-107,  Contractor  Performance 
Assessment  Reporting  System  (CPARS).  A  relationship  to  this  regulation  was  deemed 
appropriate  in  that  this  regulation  best  provides  guidance  for  implementation  of  past 
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performance  policies  (even  though  it  applies  to  acquisitions  over  $5  million).  This 
section  will  describe  how  each  interview  question  was  selected. 

A.  Storing  the  Information. 

Question  #1:  What  exactly  is  being  stored  in  your  past  performance  files? 

Question  #2:  Is  the  information  stored  manually  or  in  electronic  form? 

These  questions  were  selected  to  find  learn  how  the  AFMC  work  centers  are 
storing  their  past  performance  information.  Through  discussions  with  the  HQ  AFMC 
policy  officials  and  with  other  experienced  contracting  personnel,  it  was  anticipated  to 
find  either  an  “in-house”  database  or  predominant  use  of  questionnaires  as  a  means  of 
storing  past  performance  information. 

Question  # 3 :  How  often  is  past  performance  information  input  into  a 
contractor’s  file? 

Question  #4:  How  often  is  the  information  updated? 

Question  #5:  How  often  is  the  information  removed? 

Questions  #3  -  #5  were  asked  to  discern  the  age  of  information  that  is  used  for 
past  performance  evaluations  in  the  $1  million  to  $5  million  acquisition  range.  These 
questions  will  be  compared  with  the  CPARS  regulation. 


Question  46: 
Question  #7: 
Question  # 8 : 

Question  #9: 


How  are  contractor  name  changes  handled? 

How  are  contractor  mergers  handled? 

Is  there  a  program  manager  summary  similar  to  that  of  the 
CPAR  form? 

Are  there  other  actions  necessary  to  store  past  performance 
information? 
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These  questions  were  selected  to  answer  other  storage  issues.  Name  changes  and 
mergers  are  expected  to  occur  during  this  acquisition  range  so  it  is  necessary  to  learn  how 
the  AFMC  work  centers  are  handling  these  events  with  regards  to  storing  their  past 
performance  information.  The  program  manager  summary  was  a  key  change  to  the 
CPARS  form  for  the  Air  Force’s  Lightning  Bolt  #6  so  it  is  interesting  to  note  if  the  same 
change  was  reflected  for  acquisitions  under  $5  million.  Question  #9  was  added  to  offer 
the  opportunity  to  the  interviewees  for  any  further  comments.  It  is  an  open-ended 
question  with  the  purpose  of  extracting  data  may  not  necessarily  have  been  mentioned  in 
response  to  Questions  #1  -  #9. 

B.  Protecting  the  Information. 

Question  M 0:  What  measures  are  done  to  ensure  proper  protection  of  the 
information? 

Question  Ml:  Who  is  authorized  to  input  information  in  a  contractor ’s 
file? 

Question  M2:  Who  is  authorized  to  retrieve  information  from  a  file? 

These  questions  address  the  general  protection  methods  implemented  by  the 
various  AFMC  work  centers  as  well  as  the  specific  authorizations  provided  to  those  who 
input  and/or  retrieve  past  performance  information. 

Question  M3:  What  is  your  policy  towards  another  work  center  asking 

you  for  information  on  a  particular  contractor? 

Question  #13  was  included  for  two  reasons:  1)  To  test  if  information  sharing 
exists  between  the  work  centers  as  it  is  expected  that  no  single  work  center  can  feasibly 
contain  past  performance  information  on  all  prospective  contractors;  and  2)  To 
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understand  the  nature  of  their  responses  towards  another  work  center  requesting  past 
performance  information. 

Question  #14:  Does  a  contractor  have  the  opportunity  to  review  the  file? 

Asking  the  work  centers  if  they  provide  their  contractors  an  opportunity  to  review 
the  file  gains  an  understanding  of  whether  they  prefer  protection  methods  over  the  value 
of  more  accurate  information.  The  CPARS  regulation  mandates  an  opportunity  must  be 
provided  to  the  contractors  for  rebuttal  purposes.  However,  the  decision  is  unguided  for 
acquisitions  below  $5  million.  Some  work  centers  may  provide  the  rebuttal  opportunity 
while  other  may  prefer  to  keep  stricter  protection  methods. 

C.  Performing  Relevancy  Determinations. 

Question  #15:  For  what  kinds  of  requirements  are  you  keeping 
information? 

Before  asking  the  work  centers  about  their  relevancy  criteria  and  information 
needs,  it  is  necessary  to  test  if  they  have  similar  contract  requirements.  If  they  do,  then 
comparisons  will  be  more  effective.  If  they  do  not  have  similar  contracts  then,  logically, 
dissimilar  contract  requirements  implies  the  possibility  of  dissimilar  relevancy  criteria 
and  information  needs. 

Question  #16:  What  criteria  do  you  use  in  performing  a  relevancy 
determination? 
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Question  #16  is  the  focus  of  Objective  #3:  Performing  Relevancy 
Determinations.  It  is  necessary  to  discover  the  most  useful  relevancy  criteria  for  the 
acquisition  range  of  $1  million  to  $5  million. 

Question  #1  7:  What  challenges  do  you  have  in  performing  relevancy 
determinations? 

This  next  question  is  open-ended  and  is  included  for  the  purposes  of  gaining  the 
work  centers’  opinions  about  their  relevancy  needs.  By  identifying  the  challenges  they 
face  when  performing  relevancy  determinations,  an  understanding  is  provided  of  the 
peripheral  issues  surrounding  the  decision  to  maintain  a  database  or  gather  past 
performance  information  through  other  means. 

D.  Sources  of  Past  Performance  Information. 

Questions  #18  -  #29  depict  the  variety  of  past  performance  information 
sources  as  discussed  in  Chapter  2.  Multiple  literary  sources  were  used  to  develop  this 
collage. 

E.  Past  Performance  Information  Types. 

Questions  #30  -  #42  combine  two  models  of  past  performance  information 
into  a  mutually  exhaustive  collection  of  information  types.  This  combination  is  derived 
from  Dr.  Kelman’s  list  and  the  categories  found  in  the  CPARS  form. 
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Content  Validity 


The  interview  questions  found  in  Attachment  A  have  been  selected  to  satisfy 
concerns  of  HQ  AFMC,  confirm  or  reject  propositions  of  previous  literature,  and  to  learn 
about  the  differences  in  past  performance  information  systems.  The  questions  found  in 
“Storing  the  Information”  and  “Protecting  the  Information”  were  raised  during 
discussions  with  the  sponsor.  The  questions  in  “Performing  Relevancy  Determinations” 
were  selected  to  learn  more  about  relevance  (very  little  has  been  written  about  it)  and  to 
test  if  there  are  differences  in  relevancy  needs  among  various  information  systems. 
“Sources  of  Past  Performance  Information”  and  “Past  Performance  Information  Types” 
are  necessary  sections  to  discover  the  useful  sources  and  types  of  past  performance 
information. 

Reliability  surfaces  as  an  issue  through  proper  selection  of  the  categories  used  in 
content  analysis  (Silverman,  1993:148).  In  other  words,  “These  categories  should  be 
used  in  a  standardized  way,  so  that  any  researcher  would  categorize  in  the  same  way” 
(Silverman,  1993:148).  The  categories  related  to  the  sources  of  past  performance 
information  were  derived  from  a  consensus  of  research  literature  that  outlined  these 
categories.  The  categories  of  past  performance  information  types  were  derived  from  a 
combination  of  categories  found  in  AFMC  Instruction  64-107,  Contractor  Performance 
Assessment  Reporting  System  (the  regulatory  guidance  for  acquisitions  over  $5  million) 
and  in  Dr.  Reiman’ s  (OFPP  Administrator  of  Federal  Procurement  Policy)  past 
performance  guidebook.  As  for  the  questions  in  the  other  objectives  (#2,  #3,  and  #5), 
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they  came  from  discussions  with  HQ  AFMC  past  performance  experts,  the  CPARS 
regulation,  and  other  literature  research. 

Summary 

As  no  guidance  presently  exists  for  the  implementation  of  past  performance 
information  systems  and  its  surrounding  policies,  it  is  hypothesized  that  some  work 
centers  have  implemented  their  own  “in-house”  database  to  collect  and  maintain  the 
information.  From  the  decision  to  store  information  “in-house”  stems  the  decisions 
regarding  protection.  It  is  the  second  objective  that  ascertains  the  authorization  of 
inputting  and  retrieving  information.  Once  past  performance  information  is  stored  and 
protected,  relevancy  is  a  filter  that  determines  if  it  will  be  used  in  subsequent 
acquisitions.  Objective  #3  determines  the  major  relevancy  criteria  that  are  used  by 
AFMC  work  centers  that  affects  the  information.  The  next  two  sections,  Objective  #4 
and  #5,  reveal  the  sources  and  types  of  information  that  are  being  used  as  it  is  imperative 
to  know  from  where  the  information  comes  and  what  it  looks  like.  While  others  may  not 
have  constructed  an  “in-house”  database,  it  may  be  implied  from  this  variance  that 
similarities  and  differences  exist  in  the  approaches  surrounding  past  performance 
information.  It  is  Objective  #6  that  performs  tests  to  see  if  subsequent  similarities  and 
differences  exist  among  different  levels  of  protection,  application  of  relevancy  criteria, 
and  use  of  other  information  sources  and  past  performance  information  types. 

Interviews  provided  the  data  necessary  for  these  objectives.  The  interview 
respondents  included  hand-picked  contracting  personnel  by  HQ  AFMC.  These  subject- 
matter  experts  have  sufficient  knowledge,  expertise,  and  experience  to  offer  an  account  of 
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their  work  centers’  implementation  of  past  performance  procedures.  By  pooling  the 
interview  transcripts,  an  analysis  will  provide  similar  and  contrasting  methods  with 
AFMC.  As  a  result  of  this  analysis,  the  decision  on  whether  to  create  a  past  performance 
database  or  not  will  be  better  understood. 
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IV.  Results 


Introduction 

Eleven  contracting  professionals  participated  in  this  research  effort.  All  of  these 
people  are  Government  civilian  employees  with  five  to  twenty  years  of  contracting 
experience.  About  half  of  these  people  serve  as  the  Contractor  Performance  Assessment 
Reporting  System  (CPARS)  focal  point.  All  interviewees  are  knowledgeable  about  how 
past  performance  policies  have  been  implemented  within  their  work  center.  They 
represent  seven  different  AFMC  work  centers.  Two  product  centers  and  five  logistic 
centers  provided  the  data  necessary  for  research  in  past  performance  policies  for  the  $1 
million  to  $5  million  acquisition  range.  Each  of  the  contracting  personnel  were 
interviewed  in  accordance  with  Appendix  A.  Their  answers  are  tabulated  by  work  center 
according  to  each  interview  question  listed  in  this  attachment. 

Objective  #1 :  Storing  the  Information 

In  the  absence  of  the  Contractor  Performance  Assessment  Reporting  System 
(CPARS)  to  store  contract  information  under  $5  million,  the  work  centers  may  have 
created  their  own  “in-house”  database.  However,  no  regulatory  guidance  exists 
surrounding  this  decision  and  the  decisions  related  to  its  maintenance.  This  objective 
describes  the  various  commonalities  and  differences  in  the  methods  that  the  contracting 
professionals  have  been  using  to  store  the  information.  Out  of  seven  work  centers,  four 
have  created  such  a  database  while  the  other  three  centers’  method  is  predominant  use  of 
questionnaires  for  each  acquisition.  Table  1  displays  the  various  work  centers’ 
approaches  to  a  past  performance  information  system  as  the  results  of  Questions  #1  -  #3: 
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Table  1.  Past  Performance  Information  Systems  Used  By  AFMC  Work  Centers 


Organization 

Method  of  Information  System 

Manual  or 
Electronic 

A,  B,  and  C 

Use  questionnaires  predominantly  and  do 
not  have  a  database. 

Manual. 

D 

Has  a  repository  of  quality  assurance 
reports  and  other  records  of  contractor 
performance. 

Manual. 

E 

Has  an  unofficial  CPARS-formatted 
database. 

Electronic 

F 

Keeps  records  of  complaints  or  good 
comments  about  current  contractors  at  the 
operational  level  and  a  database  of 
contract  history  at  the  central  procurement 
level  (but  this  is  used  primarily  for 
responsibility  determinations). 

Manual  at 
operational 
level  but 
electronic  at 
central. 

G 

Maintains  a  database  of  contract  history 
(but  this  is  used  primarily  for 
responsibility  determinations). 

Electronic. 

In  summary  of  Table  1,  all  seven  work  centers  generate  new  information  for  each 
source  selection  or  exercise  of  an  option.  (Three  work  centers  send  new  questionnaires 
for  each  new  acquisition  as  their  predominant  means  of  collecting  past  performance 
information.)  Four  other  work  centers  gather  new  information  anytime  it  becomes 
available,  as  in  the  case  of  quality  assurance  surveillance  reports  (e.  g.,  deficiency  reports 
or  delivery  reports). 

Objective  #1  also  describes  how  often  information  is  input,  updated,  or  removed. 
As  shown  below,  those  work  centers  who  maintain  a  database  input  information  at  about 
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the  same  frequency,  but  they  differ  among  each  other  in  how  often  they  update  or  remove 

it.  The  following  results  are  displayed  under  each  interview  question  pertaining  to 

storing  the  information  (i.  e.,  the  questions  are  from  Appendix  A): 

Question  #4:  How  often  is  the  information  updated? 

Three  work  centers  do  not  update  their  information  because  they  generate  new 
questionnaires  for  each  acquisition.  One  work  center  undergoes  an  annual  review  of  the 
information.  Another  work  center  undergoes  a  5-6  year  update.  Another  work  center 
updates  new  information  with  each  new  acquisition.  Another  work  center  did  not 
comment. 

Question  #5:  How  often  is  the  information  removed? 

This  is  not  applicable  for  those  that  predominantly  rely  on  new  questionnaires  for 
each  acquisition.  Two  work  centers  do  not  remove  the  information.  Another  work  center 
removes  information  at  a  5-6  year  cycle.  Another  work  center  did  not  comment. 

Question  #6:  How  are  contractor  name  changes  handled? 

Four  of  the  seven  work  centers  answer  “Not  Applicable”  (three  of  these  four 
predominantly  use  questionnaires).  Two  work  centers’  automated  databases  have  a 
mechanism  to  recognize  name  changes.  Another  work  center  tries  to  isolate  key 
individuals. 

Question  #7:  How  are  contractor  mergers  handled? 

Three  of  the  seven  work  center  answer  “Not  Applicable”  (two  of  these  three 
predominantly  use  questionnaires).  Two  work  centers’  automated  databases  have  a 
mechanism  to  recognize  mergers.  Two  other  work  center  tries  to  isolate  key  individuals. 

Question  # 8 :  Is  there  a  program  manager  summary  similar  to  that  of  the  CPAR  form? 

Four  work  centers  find  this  not  necessary  (three  of  these  four  predominantly  use 
questionnaires).  Of  three  work  centers  who  maintain  performance  information  in  some 
form  of  repository,  two  include  a  program  manager  summary  but  one  does  not. 

Question  #9:  Are  there  other  actions  necessary  to  store  past  performance  information? 

1 .  Two  work  centers  would  like  to  implement  a  CPARS-formatted  database  but 
they  currently  do  not  have  one. 
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Three  of  the  seven  work  centers  responded  to  this  question  with  their 
challenges: 

2.  Information  is  not  updated  enough; 

3.  The  system  does  not  allow  for  buyer  or  program  manager  inputs; 

4.  There  must  be  a  mechanism  to  keep  the  information-gathering  process 
standardized; 

5.  They  are  unable  to  find  a  centralized  location  to  archive  past  performance 
information; 

6.  They  do  not  have  the  time  to  record  and  store  past  performance  information; 

7.  Contractors  in  general  are  expecting  the  use  of  the  CPARS  system  and  they 
are  surprised  to  find  that  CPARS  will  not  be  used  for  their  source  selections; 
and 

8.  Contractors  in  general  are  complaining  about  too  much  variance  from  one 
organization’s  proposal  to  another. 
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Objective  #2:  Protecting  the  Information 

The  CPARS  regulation  specifies  that  past  performance  information  must  be  kept 

confidential,  but  what  are  the  general  protection  methods  for  acquisitions  under  $5 
million?  This  is  Question  #10  from  the  interview.  Figure  5  shows  how  AFMC  work 
centers  protect  their  collection  of  past  performance  information: 


Manual  Repository 
Under  Lock/Key 
29% 


T  reats 

Questionnaires 
IAW  Appendix  BB 
28% 


Maintains 

Automated 

Database 

29% 


Does  Not  Exercise 
Any  Measures 
14% 


Figure  5.  Protection  Measures  Implemented  By  AFMC  Work  Centers 


Two  of  the  three  work  centers  who  predominantly  rely  on  questionnaires  treat  the 
questionnaires  in  accordance  with  source  selection  procedures  as  per  Appendix  BB.  One 
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of  these  work  centers  does  not  exercise  any  protection  measures.  Access  to  the  two  work 
centers’  automated  databases  is  only  through  user  identification  and  password.  Two 
other  work  centers  keep  a  manual  repository,  and  this  location  is  inside  a  source  selection 
building  or  room,  under  lock  and  key. 

Questions  #1 1  and  #12  (Who  has  access  to  input  information?;  and  Who  has 
access  to  retrieve  information?)  are  necessary  to  identify  further  specific  protection 
methods.  Figures  6  and  7  show  the  level  of  authorization  AFMC  work  centers  provide  to 
input  and  retrieve  past  performance  information: 


Questionnaires 
Are  Limited  to 
PRAG 
43% 


Single  Focal  Point 
14% 


Figure  6.  Authorization  to  Input  Information  By  AFMC  Work  Centers 
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For  the  three  work  centers  who  predominantly  rely  on  questionnaires,  there  is  no 
need  to  “input”  information  per  se.  Authorization  to  gather  completed  questionnaires  is 
given  to  the  members  of  the  respective  PRAG  teams.  One  of  the  two  work  centers  with 
an  automated  database  uses  a  single  focal  point  while  the  other  work  center  authorizes 
input-access  to  contracting  officers  and  buyers.  The  other  two  work  centers  who 
maintain  a  manual  repository  limit  input  access  to  the  appropriate  contracting  officers  and 
buyers  of  each  respective  contract.  One  of  these  two  work  centers  also  allows  quality 
assurance  personnel  to  input  information. 


Limited  to  CO’s 


Point 

29% 


Figure  7.  Authorization  to  Retrieve  Information  By  AFMC  Work  Centers 


For  the  three  work  centers  who  predominantly  rely  on  questionnaires,  access  is 
limited  to  the  PRAG  team  only.  Two  of  the  remaining  four  who  have  an  “in-house” 
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database  limit  access  to  a  single  focal  point.  The  other  two  other  work  centers  limit 
access  to  contracting  and  buyers  on  a  “Need  To  Know”  basis. 


Question  #13:  What  is  your  policy  towards  another  work  center  asking  you  for 
information  on  a  particular  contractor? 

Two  work  centers  will  provide  a  questionnaire  to  the  inquiring  work  center  and 
one  will  provide  a  CPARS-format  report  (with  the  caveat  that  their  report  is  unique  to 
their  own  rating  system).  Three  of  the  seven  work  centers  report  this  has  never  happened 
but  will  respond  to  the  questionnaires. 

Question  #14:  Does  a  contractor  have  the  opportunity  to  review  the  file? 

Five  out  of  the  seven  work  centers  allow  the  contractor  the  opportunity  to  review 
the  information  in  one  way  or  another.  Three  work  centers  offer  the  opportunity  by  way 
of  debriefings.  Another  work  center  offers  the  opportunity  at  the  time  the  report  is 
written.  One  work  center  provides  the  review  opportunity  anytime  a  written  request  is 
made. 


Objective  #3:  Performing  Relevancy  Determinations 

After  past  performance  information  is  stored  and  protected,  relevancy  is  the  filter 
that  determines  if  it  is  useful  to  a  particular  acquisition.  This  objective  establishes  which 
criteria  are  useful  in  performing  relevancy  determinations  for  the  lower  dollar 
acquisitions.  Question  #1 5’s  results  indicate  that  each  of  the  AFMC  work  centers  handle 
the  same  type  of  contracts  (i.  e.,  services,  repair,  spares,  etc.)  except  two  work  centers 
also  do  production  contracts. 
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Question  #7(5:  What  criteria  do  you  me  in  performing  a  relevancy  determination? 
Figure  8  displays  the  criteria  elicited  by  the  interview  respondents: 


Division  of  the  Company 
Military  or  Commercial 
Complexity 

Length  of  the  Contract 

A  Trend  of  Poor 
Performance 

Contract  Dollar  Value 

User's  Input  of  Important 
Technical  Areas 

Age  of  the  Information 
Key  Individual 
Like  or  Similar  Service 


0  1  2  3  4  5  6  7 

#  of  Work  Centers  Who  Feel  the  Criterion  is  Useful 


Figure  8.  Relevancy  Criteria  Used  By  AFMC  Work  Centers 

As  shown  in  Figure  8,  all  of  the  work  centers  agree  that  Like  or  Similar  Service  is 
a  useful  relevancy  criterion.  About  half  of  the  work  centers  agreed  that  these  are  the  four 
major  criteria  useful  in  a  relevancy  determination:  Like  or  Similar  Sendee.  Key 
Individual,  Age  of  the  Information,  and  User’s  Input  of  Important  Technical  Aspects. 
Other  criteria  were  mentioned  (e.  g.,  Contract  Dollar  Value,  Trend  of  Poor  Performance, 
Length  of  the  Contract,  etc.)  but  they  are  not  as  prominent. 


56 


There  are  challenges  involved  in  collecting  and  using  relevant  information  for  the 
past  performance  evaluations.  These  challenges  help  to  understand  the  decision  to 
implement  a  database  or  to  gather  past  performance  information  through  other  means. 
Question  #1 1:  What  challenges  do  you  have  in  performing  relevancy  determinations? 

1 .  Five  of  the  seven  work  centers  mention  the  lack  of  good  business  judgment 
practiced  by  the  contracting  officers  and  buyers. 

The  following  answers  were  each  reported  by  only  one  work  center: 

2.  Some  contractors  have  a  narrow  scope  of  performance  history; 

3 .  The  users  change  their  preferences  and  this  affects  the  relevancy  to  the 
requirements; 

4.  A  negative  bias  exists  because  when  past  performance  information  is 
usually  gathered  it  is  based  on  poor  performance; 

5.  A  communication  gap  exists  between  terminology  differences  from  one 
contracting  organization  to  another; 

6.  People  tend  to  concentrate  on  one  particular  aspect  of  negative  information; 

7.  The  CPARS  format  is  too  rigid  and  not  conforming  to  their  requirements; 

8.  Coded  information  has  been  tailored  to  each  work  center’s  priorities; 

9.  It  is  difficult  to  find  information  not  older  than  three  years; 

1 0.  It  is  difficult  to  find  information  on  a  contractor  who  has  not  had  a 
Government  contract  for  a  long  period  of  time; 

11.  It  is  difficult  to  out-guess  the  GAO’s  decisions  among  protests;  and 

12.  Documentation  of  decisions  is  poorly  performed. 
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Objective  #4:  Sources  of  Past  Performance  Information 

Before  establishing  the  information  used  in  performance  evaluations,  the  sources 

first  need  to  be  identified.  It  is  hypothesized  each  information  source  is  useful  to  some 
but  not  all  work  centers  in  collecting  past  performance  information.  All  seven  work 
centers  use  the  Contractor  Performance  Assessment  Reporting  System  (CPARS)  and 
questionnaires  and  interviews  but  not  one  of  the  centers  use  the  Contractor  Performance 
Evaluation  Program  (CPEP)  nor  the  Contractor  Profile  System/Contractor  Information 
Service  (CPS/CIS).  The  other  information  sources  vary  in  their  usefulness.  This  is 
presented  below  as  the  results  of  Questions  #18  -  #29.  (The  question  was:  “Answer  ‘Yes 
or  No  ’  if  you  currently  consider  these  information  sources  to  obtain  past  performance 
information ”) 

Contractor  Performance  Assessment  Reporting  System  (CPARS) — for  contract 
information  over  $5  million 

Yes  No  Remarks  Made  By  Interviewees 

7  0  But  CPARS  usually  does  not  contain  data 

on  the  offerors  at  hand. 

“In-house”  Performance  Reports 

Yes  No  Remarks  Made  By  Interviewees 

4  3  Only  used  if  they  are  aware  that  the 

acquisition  is  similar  to  an  existing 
record. 
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Offerors’  Proposals 


Yes  No 


7  0 


Financial  Reports 

Yes  No 

1  6 


Remarks  Made  By  Interviewees 


One  work  center  differs  from  the  rest  in 
that  they  require  the  offeror  to  obtain 
completed  questionnaires,  thus  reducing 
Government  man-hours. 


Remarks  Made  By  Interviewees 


Financial  reports  are  more  suitable  for 
responsibility  determinations  than  past 
performance  evaluations. 


Contractor  Performance  Evaluation  Program  (CPEP) 

Yes  No  Remarks  Made  By  Interviewees 


0 


7 


They  do  not  know  what  the  CPEP  is. 


Acquisition  Management  Information  System  (AMIS) 

Yes  No  Remarks  Made  Bv  Interviewees 

3  4  Some  work  centers’  computers  interface 

with  AMIS  while  others  do  not. 


Legal  Databases  (such  as  LEXIS  or  WESTLAW) 


Yes  No 


Remarks  Made  Bv  Interviewees 


0  7 


Legal  databases  are  useful  for  the  JAG 
review  but  not  for  past  performance 
evaluations. 
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Contractor  Profile  System  (CPS)  or  the  Contractor  Information  Service  (CIS) 

Yes  No  Remarks  Made  By  Interviewees 


0  7  While  one  work  center  uses  the  CPS/CIS 

for  responsibility  determinations,  the 
others  do  not  know  what  this  is. 


Contractor  Alert  List  (CAL) 

Yes  No  Remarks  Made  By  Interviewees 


4  3  Three  use  the  CAL  for  past  performance 

evaluations  and  one  uses  it  for 
responsibility  determinations. 


Other  Agency  Databases 

Yes  No  Remarks  Made  By  Interviewees 

2  5  Two  work  centers  have  found  an  Army 

construction  contract  database  useful  in 
past  performance  evaluations. 


Questionnaires  and  Interviews 

Yes  No 

7  0 


Other  Sources 

Yes  No 

2  5 


Remarks  Made  By  Interviewees 

Three  of  the  six  work  centers  use  these  as 
primary  means  of  obtaining  past 
performance  information. 


Remarks  Made  By  Interviewees 

Other  sources  include  a  database  on  the 
delivery  of  spares,  the  Debarred  List,  a 
List  of  Contractors  Indebted  to  the  United 
States,  and  Dunn  &  Brad  Street’s  supplier 
evaluation  ratings. 
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Objective  #5:  Past  Performance  Information  Types 

This  objective  explores  which  information  types  are  useful  among  AFMC  work 
centers.  It  answers  the  question:  What  kinds  of  past  performance  information  are  used  in 
evaluations?  It  is  hypothesized  that  the  work  centers  use  the  same  past  performance 
information  types  as  categorized  in  the  CPAR  format  combined  with  Dr.  Kelman’s  (who 
is  the  OFPP  Administer  of  Federal  Procurement  Policy)  list  of  information  types.  Two 
information  types,  Technical  Performance  of  the  Product  and  Unique  Aspects  (those 
tailored  exclusively  for  an  acquisition)  are  useful  to  all  seven  work  centers.  The  other 
information  types  are  useful  to  some  but  not  all  work  centers.  This  is  presented  below  as 
the  results  of  Questions  #30  -  42.  (The  question  was:  Answer  ‘Yes  or  No’ if  you 
currently  consider  these  information  types  in  obtaining  past  performance  information’’’) 


Cost  Control 

Yes  No 

5  2 

Timeliness  (Management  Responsiveness) 

Yes  No 

4  3 


Remarks  Made  By  Interviewees 

The  two  work  centers  who  disagreed  do 
not  conduct  many  cost-type  contracts. 


Remarks  Made  Bv  Interviewees 

One  work  center  who  disagreed  felt  that 
this  type  of  information  is  too  subjective 
and  therefore  not  useful. 


61 


Timeliness  (Schedule  Control) 


Yes  No 


6  1 


Business  Relations 

Yes  No 

5  2 


Remarks  Made  By  Interviewees 

Because  of  the  recurring  and  on-going 
nature  of  service  contracts,  a  work  center 
disagreed  as  to  the  usefulness  of  schedule 
control  (i.  e.,  delivery  schedule 
compliance  is  not  a  factor). 


Remarks  Made  By  Interviewees 

One  work  center  who  disagreed  felt  that 
business  relations  are  useful  in  general 
considerations,  but  should  be  considered 
with  caution  because  the  information 
could  be  the  result  of  a  personality 
conflict. 


Customer  Satisfaction  (Technical  Performance  of  Product) 

Yes  No  Remarks  Made  By  Interviewees 

7  0  All  work  centers  agreed  that  the  end- 

user’s  satisfaction  is  always  important. 

Customer  Satisfaction  (Product  Assurance  [Quality  System]) 

Yes  No  Remarks  Made  Bv  Interviewees 


6 


The  one  work  center  who  disagreed 
stated  that  this  information  type  is  not 
relevant  to  their  type  of  work. 


62 


Customer  Satisfaction  (Product  Assurance  [Manufacturing  Management]) 

Yes  No  Remarks  Made  By  Interviewees 


2  5  The  two  work  centers  who  agreed  stated 

that  this  information  type  is  relevant  to 
their  type  of  work  (i.  e.,  service 
contracts). 

Customer  Satisfaction  (Logistics  Support/Sustainment) 

Yes  No  Remarks  Made  By  Interviewees 

2  5  Two  work  centers  stated  that  this 

information  type  is  relevant  to  their  type 
of  work. 


Customer  Satisfaction  (Engineering) 

Yes  No 

2  5 


Remarks  Made  By  Interviewees 

Most  work  centers  do  not  have 
production  contracts  in  this  dollar  range. 


Key  Personnel  (Program/Data  Management) 

Yes  No  Remarks  Made  By  Interviewees 

4  3  Program  management  is  important  for 

those  work  centers  who  feel  problem¬ 
solving  is  essential  to  successfully 
complete  their  contracts. 

Key  Personnel  (Procurement/Subcontract  Management) 

No  Remarks  Made  By  Interviewees 

1  None. 


Yes 

6 
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Unique  Aspects 

Yes  No  Remarks  Made  By  Interviewees 

7  0  All  work  centers  utilize  uniquely-tailored 

information  relevant  to  the  current 
contract  requirements. 


Other  Aspects — not  already  mentioned 

Yes  No  Remarks  Made  By  Interviewees 

2  5  In  addition  to  the  information  types  listed 

above,  one  work  center  uses  facility 
management  in  their  past  performance 
evaluations. 


Objective  #6:  Similarities  and  Differences  Between  Past  Performance  Information 
Systems 

The  interview  results  revealed  that  about  half  of  the  work  centers  use  a  past 
performance  database  (or  repository)  and  the  other  half  does  not  have  a  database  of  any 
sort  to  store  past  performance  information.  They  still  collect  past  performance 
information  but  they  do  not  store  it  in  a  database.  The  past  performance  information  is 
stored  in  each  respective  contract  file  along  with  the  other  source  selection  documents. 
Thus,  two  general  types  of  work  centers  approach  collecting  and  storing  past  performance 
information  in  this  manner:  Those  that  who  predominantly  use  questionnaires  and  those 
who  have  an  “in-house”  database. 
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Test  A:  Do  the  information  systems  offer  the  contractor  an  opportunity  to  review 


the  information? 

Overall,  five  out  of  the  seven  work  centers  offer  the  contractor  an  opportunity  to 
review  the  past  performance  information.  Two  of  the  four  work  centers  who  have  a 
database  offer  the  contractor  access  to  the  file  prior  to  award.  In  contrast,  the  three  work 
centers  who  predominantly  use  questionnaires  offer  this  review  opportunity  during 
discussions. 

Test  B:  Do  the  information  systems  provide  the  same  level  of  authorization  to 

input  or  retrieve  information? 

Both  of  the  information  systems  do  provide  nearly  the  same  level  of  authorization 
to  input  information.  Six  of  the  seven  work  centers  provide  the  authorization  to 
contracting  officers  and  buyers.  (One  of  these  six  work  centers  extends  the  authorization 
to  quality  assurance  personnel.)  On  the  other  hand,  one  of  the  work  centers  who  has  a 
database  only  provides  authorization  to  a  single  focal  point. 

Both  of  the  information  systems  do  not,  however,  provide  the  same  level  of 
authorization  to  retrieve  the  information.  By  definition,  the  three  work  centers  who 
predominantly  use  questionnaires  provide  retrieval  authorization  to  the  contracting 
officers  and  buyers.  In  contrast,  two  out  of  the  four  work  centers  who  have  a  database 
provide  retrieval  authorization  to  the  contracting  officers  and  buyers  whereas  the  other 
half  provides  it  to  a  single  focal  point. 


65 


Test  C:  Do  the  information  systems  use  the  same  relevancy  criteria? 


Table  2.  Similarities  and  Differences  of  Relevancy  Criteria  By  Information  System 


Relevancy  Criteria 

Like  or  Similar  Sendee 

Key  Individual 

Age  of  the  Information 

User's  Input  of  Important  Technical  Aspects 

Contract  Dollar  Value 

Trend  of  Poor  Performance 


%  of  Those  Who 
Predominantly  Use 
Questionnaires 
mo  Find  This 
Criterion  Useful 
100 
66 
66 
66 
33 
0 


%  of  Those 
Who  Have  a 
Database  mo 
Find  This 
Criterion  Usei 
100 
SO 
25 
25 
25 
50 


%  of  Work  Centers  Who  Find  Criterion  Useful 


(Legend:  Top  =  Database.  Bottom  =  Predominant  Use  of  Questionnaires) 
Figure  9.  Similarities  and  Differences  of  Relevancy  Criteria  By  Information  System 
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All  work  centers  answered  that  Like  or  Similar  Service  is  useful  in  a  relevancy 
determination.  As  shown  in  Figure  9,  both  types  of  work  centers  reported  in  the  same 
proportion  that  the  Key  Individual  is  a  useful  relevancy  criteria  (Q  =  66%  vs.  D  =  50%). 
In  similar  fashion,  not  as  many  work  centers  reported  that  the  Contract  Dollar  Value  is 
useful  (Q  =  33%  vs.  D  =  25%).  However,  those  work  centers  who  predominantly  use 
questionnaires  reported  Age  of  the  Information  and  User’s  Input  of  Important  Technical 
Aspects  more  than  those  work  centers  who  have  a  database  (for  both  criteria,  Q  =  66%  vs. 
D  =  25%).  Conversely,  those  who  have  a  database  reported  more  use  of  Trend  of  Poor 
Performance  than  those  who  predominantly  use  questionnaires  (D  =  50%  vs.  Q  =  0%). 


Test  D:  Do  all  work  centers  use  the  same  sources  of  past  performance 
information  regardless  if  they  have  an  in-house  database  or  not? 

Table  3.  Similarities  and  Differences  of  Past  Performance  Information  Sources  By 

Information  System 


%  of  Those  Who 

%  of  Those 

Predominantly  Use 

Who  Have  a 

Questionnaires 

Database  Who 

Who  Find  This 

Find  This 

Sources  of  Past  Performance  Information 

Source  Useful 

Source  Useful 

CPARS 

100 

100 

“In-house”  Performance  Reports 

0 

100 

Offeror’s  Proposals 

100 

100 

Financial  Reports 

0 

25 

Contractor  Performance  Evaluation  Program 

0 

0 

Acquisition  Management  Information  System 

66 

25 

Legal  Databases 

0 

0 

CPS/CIS 

0 

0 

Contractor  Alert  List 

100 

25 

Other  Agency  Databases 

0 

50 

Questionnaires  and  Interviews 

100 

100 

Other  Sources  (Contractor  Responsibility 

0 

50 

Reporting  Program) 
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CRRP 
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Other  Agency  Databases 

CAL 
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%  of  Work  Centers  Who  Find  Source  Useful 


Figure  10.  Similarities  and  Differences  of  Past  Performance  Information  Sources  By 

Information  System 

All  work  centers  use  CPARS,  offeror's  proposals,  and  questionnaires  and 
interviews.  Test  D  presents  similarity’  in  that  all  work  centers  do  not  use  these  following 
information  sources:  Contractor  Performance  Evaluation  Program,  legal  databases,  and 
Contractor  Profile  System/ Contractor  Information  Service  for  past  performance 
evaluations.  The  work  centers  contrast  in  that  those  work  centers  who  predominantly  use 
questionnaires  reported  use  the  Acquisition  Management  Information  System  (Q  =  66% 
vs.  D  =  25%)  and  the  Contractor  Alert  List  (Q  =  100%  vs.  D  =  25%)  more  than  those 
work  centers  who  have  their  own  database. 
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Test  E:  Do  the  information  systems  use  the  same  past  performance  information 


types? 


Table  4.  Similarities  and  Differences  of  Past  Performance  Information  Types  By 

Information  System 


%  of  Those  Who 

%  of  Those 

Predominantly  Use 

Who  Have  a 

Questionnaires 

Database  Who 

Who  Find  This 

Find  This 

Past  Performance  Information  Type 

Tvne  Useful 

Type  Useful 

Cost  Control 

100 

50 

Timeliness  (Mgt  Responsiveness) 

66 

50 

Timeliness  (Schedule  Control) 

100 

75 

Business  Relations 

66 

75 

Customer  Satisfaction  (Technical  Performance) 

100 

100 

Customer  Satisfaction  (Quality  Assurance) 

66 

100 

Customer  Satisfaction  (Manufacturing  Mgt) 

33 

25 

Customer  Satisfaction  (Logistics  Support) 

0 

50 

Customer  Satisfaction  (Engineering) 

33 

25 

Key  Personnel  (Program/Data  Mgt) 

66 

50 

Key  Personnel  (Subcontract  Mgt) 

100 

75 

Unique  Aspects 

100 

100 
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Key  Personnel 
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£  Quality  Assurance 
© 
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(Schedule  Control) 

Timeliness  (Mgt 
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0  20  40  SO  80  100 

%  of  Work  Centers  Who  Find  Information  Type  Useful 


(Legend:  Top  =  Database,  Bottom  =  Predominant  Use  of  Questionnaires) 

Figure  1 1 .  Similarities  and  Differences  of  Past  Performance  Information  Types  By 

Information  System 


Both  types  of  work  centers  reported  that  Technical  Performance  of  the  Product, 
Unique  Aspects.  Schedule  Control,  Subcontract  Management,  Quality  Assurance, 
Business  Relations,  Management  Responsiveness,  and  Program/Data  Management  are 
useful  information  types  for  past  performance  evaluations.  The  information  types. 
Manufacturing  Management  and  Engineering  were  not  as  prominent  (Q  =  33%  vs.  D  = 
25%)  among  either  type  of  work  center.  The  two  types  of  work  centers  are  different  in 
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that  those  who  predominantly  use  questionnaires  find  Cost  Control  more  useful  (Q  = 
100%  vs.  D  =  50%)  and  those  who  have  a  database  find  Logistics  Support/Sustainment 
more  useful  (D  =  50%  vs.  Q  =  0%). 

Summary 

The  results  indicate  there  are  two  basic  types  of  information  systems  to  store  past 
performance  information  for  the  $1  million  to  $5  million  contract  range.  Among  the 
various  tests  (i.  e.,  contractor  review  opportunity,  input/retrieve  authorization,  relevancy 
criteria,  information  sources  and  types),  similarities  as  well  as  differences  between  these 
information  systems  exist.  No  noticeable  differences  exist  among  those  two  types  of 
information  systems  in  offering  a  contractor  the  opportunity  to  review  the  performance 
information.  Both  of  the  information  systems  provide  nearly  the  same  level  of 
authorization  to  input  the  information  but  they  do  not  provide  the  same  authorization  to 
retrieve  it.  Differences  in  relevancy  criteria  exist  as  well.  Although  agreement  exists  on 
three  relevancy  criteria,  Like  or  Similar  Service  and  Key  Individual,  and  Contract  Dollar 
Value,  differences  exist  among  the  criteria  of  Trend  of  Poor  Performance,  User’s  Input  of 
Important  Information,  and  Age  of  the  Information.  As  for  the  sources  of  information, 
agreement  of  usefulness  exists  for  CPARS,  offeror’s  proposals,  and  questionnaires  and 
interviews.  Both  information  systems  do  not  use  the  Contractor  Performance  Evaluation 
Program,  legal  databases,  and  Contractor  Profile  System/Contractor  Information  Service. 
However,  those  who  predominantly  use  questionnaires  find  the  Acquisition  Management 
Information  Service  and  Contract  Alert  List  more  useful  than  those  who  have  their  own 
database.  Useful  to  both  types  of  work  centers  are  the  past  performance  information 
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types:  Technical  Performance  of  the  Product,  Unique  Aspects,  Schedule  Control, 
Subcontract  Management,  Quality  Assurance,  Business  Relations,  Management 
Responsiveness,  and  Program/Data  Management.  Likewise,  both  types  of  work  centers 
also  did  not  find  Manufacturing  Management  and  Engineering  very  useful.  In  contrast, 
while  Cost  Control  was  more  useful  to  one  information  system,  Logistics 
Support/Sustainment  was  more  useful  to  the  other. 
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V.  Conclusions  and  Recommendations 


Introduction 

The  past  performance  evaluation  process  is  a  method  of  selecting,  with  a  certain 
confidence,  an  offeror  that  has  the  capability  and  desire  to  accomplish  the  promises  of  the 
proposal.  It  certainly  has  the  nation’s  top-level  attention  as  the  Office  of  Federal 
Procurement  Policy,  the  Federal  Acquisition  Streamlining  Act  of  1994,  and  the  Air  Force 
Lightning  Bolts  have  officially  recognized  its  importance  in  achieving  “best  value.”  As 
regulations,  guides,  handbooks,  and  pamphlets  have  been  issued  in  regards  to  its 
implementation,  little  or  no  attention  has  been  placed  on  acquisitions  below  $5  million. 
Without  such  assistance,  those  work  centers  who  perform  acquisitions  below  $5  million 
have  made  their  own  decisions  as  to  which  information  types  are  useful  and  from  what 
sources  they  shall  come.  They  have  implemented  their  own  policies  regarding  storage 
and  protection  issues.  Among  all  of  these  decisions,  there  is  some  harmony  and  variance. 
It  is  of  interest  to  HQ  AFMC  policy  officials  to  know  what  works  well  in  the  field.  From 
this  foundation,  they  will  better  understand  the  needs  and  challenges  that  their  AFMC 
work  centers  face  for  the  $1  million  to  $5  million  acquisition  range.  Herein,  this  research 
effort  focuses  on  the  similarities  and  differences  among  the  implementation  issues  of  past 
performance. 

The  research  effort  was  broken  into  six  different  objectives.  These  objectives 
answer  the  question:  How  is  past  performance  handled  by  AFMC  work  centers  for  the  $1 
million  to  $5  million  acquisition  range?  Specifically,  the  first  five  objectives  include 
emphasis  on  storing  and  protecting  past  performance  information,  performing  relevancy 
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determinations  (i.  e.,  what  criteria  are  useful  in  doing  so),  as  well  as  useful  information 
types  and  sources.  The  final  objective  establishes  similarities  and  differences  between 
information  systems  for  each  of  the  first  five  objectives. 

Interviews  were  used  in  order  to  answer  the  questions  within  each  objective. 

They  included  pre-selected  contracting  personnel  who  have  the  requisite  experience  and 
knowledge  of  their  work  centers’  policies  regarding  past  performance.  The  transcripts 
were  coded,  tabulated,  and  analyzed  into  results. 

There  are  two  general  types  of  information  systems  that  work  centers  use  in  their 
approach  to  collecting  and  storing  past  performance  information.  They  are  either:  Those 
who  predominantly  use  questionnaires  and  those  who  have  an  “in-house”  database.  The 
difference  in  this  approach  may  imply  that  subsequent  decisions  (i.  e.,  protecting  the 
information,  relevancy,  information  sources  and  types)  differ  as  well.  If  differences  exist 
among  these  subsequent  decisions,  then  standardized  regulations  for  all  work  centers  is 
not  recommended.  The  central  question,  which  is  the  selection  of  the  right  information 
system  for  AFMC  work  centers,  is  affected  by  these  peripheral  issues.  Within  these 
peripherals,  similarities  and  differences  exist  as  well.  It  is  through  the  knowledge  of 
these  similarities  and  differences  that  one  may  determine  the  right  approach  to  collecting 
and  storing  past  performance  information. 

Objective  #1 :  Storing  the  Information 

The  hypothesis  that  some  work  centers  have  constructed  their  own  “in-house” 
database  is  true.  However,  this  decision  has  not  been  consistent  in  the  field.  Three  work 
centers  do  not  have  a  database  of  any  sort.  Of  these  work  centers,  one  is  currently 
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developing  an  electronic  database.  Of  the  remaining  four  work  centers,  one  maintains  a 
repository  of  quality  assurance  reports  and  other  records  of  contractor  performance.  Two 
of  them  maintain  an  electronic  database  that  has  been  established  for  the  recurring 
performance  history  of  delivering  spares.  Only  one  work  center  has  developed  a 
Contractor  Performance  Assessment  Reporting  System-formatted  database.  The  fact  that 
variance  exists  among  the  approach  to  storing  past  performance  information  implies  each 
work  center  is  independently  considering  their  requirements  for  handling  past 
performance  information  for  the  $1  million  to  $5  million  acquisition  range. 

Subsequently,  they  consider  the  information  type,  source,  relevancy  needs,  and  protection 
measures  in  determining  the  structure  of  their  past  performance  information  system. 

There  are  several  possible  reasons  why  some  work  centers  have  not  created  a 
database  for  the  $1  million  to  $5  million  acquisition  range.  One  reason  is  because  of  the 
investment  costs.  Another  reason,  as  one  interviewee  explained,  was  that  “hundreds  of 
offerors”  respond  to  acquisitions  in  this  dollar  range  and  they  cannot  feasibly  collect, 
update,  and  maintain  performance  records  on  all  of  these  contractors.  Three  other 
interviewees  further  comment  that  their  decision  to  not  create  a  database  is  because  it 
would  not  be  updated  enough.  One  other  interviewee  noted  that  their  obstacles  in 
establishing  a  database  are  finding  a  centralized  location  to  archive  the  information  and 
having  the  time  to  record  and  store  it.  Five  out  of  the  seven  work  centers  mentioned  that, 
through  observations  and  experiences,  their  centers’  main  challenge  was  the  lack  of 
business  judgment  on  the  part  of  some  workers  who  may  become  dependent  on 
computerized  information  to  perform  this  function.  This  is  also  the  result  of  information 
that  is  not  updated  enough  and  having  a  system  that  does  not  offer  buyer  or  program 
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manager  inputs.  Therefore,  if  a  centralized  database  was  created,  significant  investment 
costs  and  management  controls  would  be  required  to  maintain  an  effective  repository  of 
information  for  use  in  past  performance  evaluations. 

Objective  #2:  Protecting  the  Information 

Most  work  centers  regard  the  past  performance  information  with  strict  security 
(full  compliance  with  AFFARS  Appendix  BB  and  AFMCI  64-107),  while  one  work 
center  does  not  exercise  any  measures  at  all.  Just  as  they  would  for  acquisitions  above  $5 
million,  the  work  centers  feel  protection  is  imperative  below  $5  million  because  of  the 
proprietary  nature  of  contractor  information  and  the  information’s  involvement  in  a 
competitive  source  selection. 

Test  A  ('Objective  #61:  Do  the  information  systems  offer  the  contractor  an 

opportunity  to  review  the  information? 

Both  types  of  systems  provide  the  contractor  an  opportunity  for  review.  While 
two  of  the  four  work  centers  who  have  a  database  offer  the  contractor  access  to  the  file 
prior  to  award,  the  three  work  centers  who  predominantly  use  questionnaires  offer  this 
review  opportunity  during  discussions.  Therefore,  a  total  of  five  work  centers  offer  the 
review  opportunity  before  the  decision  has  been  made.  The  AFMCI  64-107  encourages 
this  practice  for  acquisitions  above  $5  million,  and  it  is  reflected  in  acquisitions  below 
this  dollar  level.  The  two  work  centers  who  do  not  offer  the  contractor  the  review 
opportunity  tend  to  have  a  more  restrictive  protection  measures.  However,  this  restrictive 
approach  does  not  motivate  a  contractor  to  improve  contract  performance  because 
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feedback  is  missing.  Without  feedback,  the  contractors  do  not  know  what  information  is 
kept  and  therefore  what  areas  need  improvement.  Feedback,  as  it  results  in  improving 
performance,  is  one  of  the  six  reasons  past  performance  information  should  be  used  in 
Department  of  Defense  source  selections  (Little,  1996:38).  In  answer  to  Test  A,  both 
types  of  information  systems  reflect  the  value  of  providing  their  contractors  an 
opportunity  to  review  their  files. 

Test  B  (Objective  #6):  Do  the  information  systems  provide  the  same  level  of 

authorization  to  input  or  retrieve  information? 

Both  of  the  information  systems  do  provide  nearly  the  same  level  of  authorization 
to  input  information  as  six  of  the  seven  work  centers  provide  the  authorization  to 
contracting  officers  and  buyers.  Only  one  of  the  work  centers  limits  authorization  to  a 
single  focal  point.  As  for  the  rest,  it  seems  that  input  authorization  is  provided  to  the 
personnel  who  are  best  able  to  document  performance  history,  regardless  of  the  type  of 
information  system.  Thus,  the  same  level  of  input  authorization  is  provided  for  both 
information  systems.  This  is  similar  to  the  input  authorization  provided  for  acquisitions 
above  $5  million. 

Both  of  the  information  systems  do  not,  however,  provide  the  same  level  of 
authorization  to  retrieve  the  information.  By  the  nature  of  the  questionnaire  process,  the 
three  work  centers  who  predominantly  use  questionnaires  provide  retrieval  authorization 
to  the  contracting  officers  and  buyers.  Half  of  the  work  centers  who  have  a  database 
provide  authorization  to  a  single  focal  point  whereas  the  other  half  extends  authorization 
to  contracting  officers  and  buyers.  A  single  focal  point  is  advantageous  to  ensure  proper 
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protection  of  the  information.  In  comparison,  it  is  the  standard  case  for  acquisitions  over 
$5  million  (AFMCI  64-107).  However,  extending  this  authorization  to  contracting 
officers  and  buyers  speeds  the  process  for  those  who  need  to  know  the  information  by 
eliminating  the  middle-man.  This  streamlining  effort  has  been  made  because  of  the  larger 
number  and  faster  acquisition  demands  that  are  placed  upon  contracting  officers  and 
buyers  for  acquisitions  below  $5  million. 

Objective  #3:  Performing  Relevancy  Determinations 

About  half  of  the  work  centers  answered  with  these  four  major  criteria  that  are 
useful  in  a  relevancy  determination:  Like  or  Similar  Service,  Key  Individual,  Age  of  the 
Information,  and  User’s  Input  of  Important  Technical  Aspects.  Other  criteria  were 
mentioned  (e.  g..  Contract  Dollar  Value,  Trend  of  Poor  Performance,  Length  of  the 
Contract,  etc.)  but  they  are  not  as  prominent.  The  criterion.  Like  or  Similar  Service,  was 
the  most  prominent  as  each  work  center  mentioned  this  answer.  This  is  not  surprising  as 
AFMC  defines  relevancy  as  “How  closely  the  skill  demonstrated  in  the  prior  contract,  e. 
g.,  subcontract  management,  matches  the  degree  to  which  that  skill  will  be  utilized  on  the 
new  contract”  (AFMCPAM  64-113,  1995:6).  The  criteria,  Key  Individual,  Age  of  the 
Information,  and  User’s  Input,  may  have  been  prominent  because  of  their  heuristic  effect 
of  narrowing  the  performance  history  down  to  manageable  pieces  of  information. 

Even  though  all  of  the  work  centers  have  the  same  major  relevancy  needs,  certain 
challenges  exist  that  demonstrate  how  independence  is  most  suitable  for  handling  past 
performance  for  acquisitions  below  $5  million.  Notice  how  the  four  major  types  of 
relevancy  criteria  are  objective  rather  than  subjective.  The  challenges  expressed  in 
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regards  to  relevancy  needs  involve  subjectivity.  One  of  the  challenges  associated  with 
subjectivity  is  the  negative  bias  involved  with  documenting  performance  (i.  e.,  if  it  is 
documented,  then  it  is  probably  for  the  purposes  of  recording  poor  performance).  Also, 
the  CPARS  format  is  too  rigid  and  its  coded  information  does  not  reflect  each  work 
center’s  priorities  and  relevancy  standards.  This  implies  that  information  for  one  work 
center  may  not  be  appropriate  for  another.  Likewise,  one  interviewee  mentioned  they 
sometimes  receive  questionnaires  that  have  little  or  no  information  value,  which  is  due  to 
the  questionnaire  respondent’s  lack  of  documentation  and/or  definitive  comments.  All  in 
all,  subjectivity  presents  a  challenge  when  dealing  with  relevancy.  Consequently, 
information  based  on  one  work  center’s  needs  may  not  necessarily  provide  any 
information  value  to  another.  Therefore,  a  database  containing  subjective  past 
performance  information  for  work-center-sharing  is  not  recommended  for  the  $1  million 
to  $5  million  acquisition  range. 

Because  one  work  center  cannot  expect  to  contain  past  performance  information 
on  all  contractors  conducting  Government  contracts  between  $1  million  and  $5  million, 
one  might  expect  the  work  centers  to  share  past  performance  information  between 
themselves.  However,  as  far  as  providing  information  from  a  database  (i.  e.,  more 
objective  information)  to  a  requesting  work  center,  reluctance  is  the  norm.  One  work 
center  will  supply  information  from  its  database  with  the  caveat  that  their  report  is  unique 
to  their  rating  system.  Three  other  work  centers  refuse  to  provide  the  information 
because  they  exercise  more  stringent  protection  measures.  From  these  results,  it  seems  as 
though  the  work  centers  prefer  to  collect  their  past  performance  information 
independently  from  each  other.  A  central  repository  may  not  be  suitable  at  all.  This  is 
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supported  as  several  respondents  indicated  they  prefer  to  know  from  where  the 
information  comes.  And  as  previously  mentioned,  they  are  even  more  reluctant  to  gather 
subjective  information. 

Test  C  (Objective  #6):  Do  the  information  systems  use  the  same  relevancy 

criteria? 

Both  information  systems  have  the  same  relevancy  requirements  of  Like  or 
Similar  Service  and  the  Key  Individual.  The  criteria,  Contract  Dollar  Value,  was  not 
prominent  among  both  types  of  systems.  However,  those  work  centers  who 
predominantly  use  questionnaires  reported  Age  of  the  Information  and  User’s  Input  of 
Important  Technical  Aspects  more  prominently  than  those  work  centers  who  have  a 
database.  Conversely,  those  who  have  a  database  reported  more  use  of  Trend  of  Poor 
Performance  than  those  who  predominantly  use  questionnaires.  Perhaps  the  reason  why 
those  who  predominantly  use  questionnaires  did  not  find  Trend  of  Poor  Performance 
useful  was  because  of  the  contractor  bias  involved  when  supplying  the  previous  contracts 
and  points  of  contact  for  use  in  subsequent  questionnaires  and  interviews.  Instead,  they 
rely  on  more  timely,  customer-driven  information.  On  the  other  hand,  databases  can 
generally  provide  more  information  than  a  questionnaire  so  a  trend,  or  track  record,  is 
easier  to  identify  with  a  database  file.  In  conclusion,  the  two  information  systems  use 
different  relevancy  criteria  in  the  $1  million  to  $5  million  acquisition  range. 
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Objective  #4:  Sources  of  Past  Performance  Information 


The  hypothesis  that  each  information  source  is  useful  to  some  but  not  all  work 
centers  in  collecting  past  performance  information  for  acquisitions  below  $5  million  is 
supported  but  with  exceptions.  (The  exceptions  are:  All  seven  work  centers  use  CPARS, 
offeror’s  proposals,  and  questionnaires  and  interviews;  and  No  work  center  uses  the 
Contractor  Performance  Evaluation  Program  nor  the  Contractor  Profile 
System/Contractor  Information  Service.)  Different  than  contracts  above  $5  million,  the 
$1  million  to  $5  million  acquisition  range  offers  the  opportunity  to  each  work  center  to 
store  past  performance  information  “in-house”  if  they  choose  to  create  a  repository  or 
database.  Four  work  centers  use  their  own  database,  but  only  if  they  are  aware  that  the 
acquisition  is  similar  to  an  existing  record.  As  to  not  having  an  “in-house”  database,  one 
interviewee  noted  that  a  database  would  not  have  a  wide  performance  history  and  that 
these  records  would  only  supplement  the  information  gathered  from  questionnaires.  The 
Acquisition  Management  Information  Service  (AMIS)  and  Contractor  Alert  List  (CAL) 
are  used  by  about  half  of  the  work  centers.  AFMC  maintains  AMIS,  so  it  is  not 
surprising  that  it  is  used  by  AFMC  work  centers.  Those  who  have  reported  use  of  the 
CAL  also  mentioned  having  a  strong  relationship  with  their  administrative  contracting 
officers,  who  belong  to  the  Defense  Contract  Management  Command  (they  maintain  this 
database)  (Biddy,  1995:12).  As  mentioned,  about  half  of  the  work  centers  use  these  two 
databases.  The  other  half  answered  that  they  are  unaware  of  these  sources.  The  CPEP 
and  CPS/CIS  are  used  by  no  AFMC  work  center  at  all  for  the  same  reason.  In  summary, 
the  various  work  centers  independently  select  their  own  combination  of  information 
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sources.  These  combinations  are  not  necessarily  the  same  for  all  work  centers.  Also, 
some  sources  are  used  by  all  seven  work  centers  but  other  sources  are  not  used  at  all. 

Test  D  (Objective  #61 :  Do  all  work  centers  use  the  same  sources  of  past 

performance  information  regardless  if  they  have  an  in-house  database  or  not? 

Regardless  of  the  predominant  use  of  questionnaires  or  having  an  “in-house” 
database  of  reports,  all  work  centers  use  CPARS,  offeror’s  proposals,  and  questionnaires 
and  interviews.  This  reflects  the  significant  information  value  of  these  sources. 

Similarly,  they  are  acclaimed  information  sources  for  Air  Force  acquisitions  above  $5 
million.  All  work  centers  are  similar  in  that  they  do  not  use  these  following  information 
sources:  the  Contractor  Performance  Evaluation  Program,  legal  databases,  and  the 
Contractor  Profile  System/Contractor  Information  Service.  As  mentioned,  legal 
databases  are  more  suitable  for  responsibility  determinations.  FAR  9.103(d)  cites  a 
responsible  contractor  is  one  that  has  a  “satisfactory  record  of  integrity  and  business 
ethics,”  which  is  the  information  indicated  in  a  legal  database.  The  work  centers  reported 
that  they  are  unaware  of  the  two  other  databases.  The  work  centers  contrast  in  that  those 
work  centers  who  predominantly  use  questionnaires  are  more  apt  to  use  the  Acquisition 
Management  Information  System  and  the  Contractor  Alert  List  than  those  work  centers 
who  have  their  own  database.  This  reflects  how  those  who  have  their  own  database  have 
the  information  more  readily  available  than  those  work  centers  who  rely  on  obtaining  the 
information  from  the  outside. 
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Objective  #5:  Past  Performance  Information  Types 


Technical  Performance,  Unique  Aspects  (tailored  to  each  acquisition),  Quality 
Assurance,  and  Schedule  Control  were  most  prominently  mentioned  because  of  their 
importance  in  satisfying  the  user.  These  types  of  information  are  the  most  objective  in 
comparison  to  the  others.  Thus,  objective  information  is  best  suited  for  the  $1  million  to 
$5  million  acquisition  range.  Business  Relations,  Management  Responsiveness,  and 
Program/Data  Management  are  next  in  prominence  because  they  are  useful  yet 
subjective.  They  are  still  useful  the  acquisition  range  but  not  so  much  as  the  more 
objective  information  types.  Those  information  types  that  scored  the  lowest  in 
comparison  to  the  others  are:  Manufacturing  Management,  Logistics 
Support/Sustainment,  and  Engineering.  These  information  types  are  only  relevant  to  two 
work  centers’  acquisitions.  As  one  respondent  noted,  the  $1  million  to  $5  million 
acquisition  range  does  not  normally  include  production  type  requirements.  It  is  was 
anticipated  that  the  CPARS  form  was  exhaustive,  but  the  data  revealed  an  information 
type  used  by  two  work  centers:  Facility  Management.  This  information  type  may  have 
emerged  as  a  result  of  the  increased  use  of  privatization  which  may  sometimes  be 
included  in  the  $1  million  to  $5  million  acquisition  range. 

Test  E  (Objective  #6):  Do  the  information  systems  use  the  same  past  performance 
information  types? 

The  information  needs  are  the  same  for  both  kinds  of  work  centers.  Useful  to 
both  types  of  work  centers  are  Technical  Performance  of  the  Product,  Unique  Aspects, 
Schedule  Control,  Subcontract  Management,  Quality  Assurance,  Business  Relations, 
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Management  Responsiveness,  and  Program/Data  Management.  Again,  both  types  of 
information  systems  prefer  objective  over  subjective  information  for  the  $1  million  to  $5 
million  acquisition  range.  Manufacturing  Management  and  Engineering  are  not  useful  to 
all  work  centers,  regardless  of  their  information  system.  The  two  types  of  work  centers 
are  different  in  that  those  who  predominantly  use  questionnaires  find  Cost  Control  more 
useful  whereas  those  who  have  a  database  find  Logistics  Support/Sustainment  more 
useful.  This  certain  polarity  is  not  due  to  the  difference  in  information  systems;  rather,  it 
is  because  of  the  types  of  contracts  they  have. 

Recommendations 

1)  The  decision  to  establish  a  database  or  to  obtain  past  performance  information 
through  other  means  should  be  derived  by  considering  if  the  information  could  first 
be  collected  through  other  effective  means.  Those  work  centers  who  predominantly 
use  questionnaires  still  use  other  databases  (e.  g.,  the  Acquisition  Management 
Information  Service  [AMIS]  and  the  Contractor  Alert  List  [CAL])  more  than  those 
who  already  have  their  own  database.  However,  even  these  and  other  databases  are 
unknown  to  the  other  work  centers.  Instead  of  creating  a  database  to  store  past 
performance  information,  using  an  already-established  database  may  eliminate 
redundant  work.  Therefore,  these  databases  (i.  e.,  the  AMIS,  and  the  CAL)  should  be 
advertised  or  introduced  to  the  various  AFMC  work  centers. 

2)  The  general  results  concerning  information  sources  indicate  that  the  AFMC  work 
centers  are  also  unaware  of  the  Contractor  Performance  Evaluation  Program  and  the 
Contractor  Profile  System/Contractor  Information  Service.  These  two  databases  are 
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unexplored  territory.  The  decision  to  create  a  database  or  rely  on  questionnaires  may 
lie  in  a  combination  of  the  CPEP,  and  the  CPS/CIS,  as  well  as  the  AMIS  and  the 
CAL.  Thus,  these  databases  need  introduction  to  the  various  AFMC  work  centers. 

3)  Other  feasibility  aspects  must  first  be  considered  before  establishing  a  database  for 
storing  past  performance  information  within  the  $1  million  to  $5  million  dollar  range. 
A  database  containing  subjective  information  for  the  purposes  of  sharing  past 
performance  information  is  not  recommended  because  of  the  relevancy  challenges 
associated  with  subjectivity  (i.  e.,  useful  and  relevant  information  for  one  work  center 
may  not  be  useful  for  another).  These  aspects  include  the  potentials  for 
communication  gaps,  negative  biases,  inflexibility,  inaccurate  documentation,  and 
out-dated  information.  Likewise,  a  database  containing  objective  information  for  the 
purposes  of  information  sharing  is  not  recommended  because  of  the  reluctance 
expressed  by  various  work  centers  (i.  e.,  differences  exist  in  documenting/coding 
relevant,  objective  information). 

4)  The  work  centers’  contracts  vary  in  the  type  of  work,  and  information  that  is  relevant 
to  one  work  center  may  not  necessarily  be  relevant  to  another.  Information  needs  and 
relevancy  criteria  differ  among  work  centers.  Therefore,  latitude  should  be  given  to 
each  work  center  as  to  which  past  performance  information  types  should  be  collected. 
Standardization  would  only  confuse  the  process  as  differences  regarding  relevance 
exist  from  one  work  center  to  another. 

5)  Those  work  centers  who  do  not  provide  their  contractors  the  opportunity  to  review 
their  file  are  facing  two  consequences.  First,  those  contractors  do  not  receive 
feedback  and  therefore  have  less  motivation  towards  improving  contract  performance 
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for  the  purposes  of  subsequent  source  selections.  Also,  those  contractors  are  more 
prone  to  protest  during  the  source  selection  as  the  rebuttal  opportunity  ensures  more 
accurate  information  is  used  during  the  evaluation  process. 

Limitations 

Due  to  the  restrictive,  protective  nature  regarding  source  selection  documents,  this 
research  effort  could  not  include  an  analysis  of  Performance  Risk  Assessment  Reports, 
actual  questionnaires  with  contractor  information,  nor  database  files  regarding  contractor 
performance.  This  kind  of  analysis  would  have  made  an  effective  triangulation  to 
ascertain  the  effectiveness  of  each  approach  regarding  the  storing,  collecting,  and 
protecting  of  past  performance  information  types  and  sources. 

It  was  not  economically  feasible  to  conduct  all  of  the  interviews  at  each 
respondents’  office.  This  limitation  acknowledges  that  it  was  not  possible  to  read  the 
same  question  in  the  same  office  environment  to  each  interviewee.  Thus,  subtle 
differences  exist  among  the  interviews.  These  differences  consist  of  background 
information  and/or  paraphrasing  after  the  interview  questions  were  read  verbatim. 
However,  the  objective  of  presenting  each  interviewee  the  same  question  was  met. 
Resultantly,  reliability  is  not  significantly  reduced. 

The  research  effort  exclusively  invoked  AFMC  work  centers  because  of  the 
relationship  with  the  sponsoring  activity,  HQ  AFMC.  The  sponsor  identified  applicable 
work  centers  who  conduct  $1  million  to  $5  million  acquisitions  and  provided  points  of 
contact.  The  Air  Force  Materiel  Command  has  unique  requirements  not  akin  to  the  rest 
of  the  Air  Force.  Thus,  the  results  from  these  seven  work  centers  do  not  necessarily 
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apply  towards  the  practices  of  the  entire  Air  Force’s  contracting  activities  regarding  past 
performance. 

Recommendations  for  Further  Research 

Part  of  the  Acquisition  Reform  push  throughout  all  acquisition  organizations  is  to 
learn  more  about  commercial  practices.  However,  this  study  did  not  include  the  defense 
industry’s  approach  to  past  performance  practices.  A  benchmarking  investigation  into 
these  approaches  is  highly  encouraged  as  it  is  important  to  compare  and  contrast  how  past 
performance  is  used  and  treated  between  these  two  sectors. 

Other  recommendations  for  further  research  are  analyzing  the  Government’s 
impact  of  past  performance  on  contractor  performance,  investigating  the  statistical 
contribution  of  past  performance  towards  the  final  award  decision,  and  comparing  and 
contrasting  the  use  of  past  performance  between  commercial-off-the-shelf  and 
developmental  acquisitions. 

Also,  with  FASA’s  mandate  to  implement  past  performance  evaluations  for 
acquisitions  above  $100,000  by  1999,  another  study  could  focus  on  the  current  processes 
and  standardizations  for  this  dollar  level. 

Summary 

Past  performance  is  a  key  element  of  selecting  contractors  with  proposals  that 
have  best  value.  Its  effectiveness  is  derivative  of  several  factors  including  the 
information  type  used  during  the  evaluation,  the  source  of  the  information,  the 
information’s  relevancy  to  the  current  acquisition,  how  well  the  information  is  protected, 
and  the  means  in  which  it  is  stored.  Much  top-level  attention  has  been  directed  towards 
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implementing  past  performance  practices  for  the  major  acquisitions.  In  fact,  the  OFPP 
mandated  the  evaluation  of  past  performance  on  all  contracts  above  $  1 00,000,  with 
incremental  implementation  stages.  As  such,  the  $1  million  to  $5  million  acquisition 
range  has  been  left  without  the  same  focus,  research,  and  efforts  as  the  over  $5  million 
acquisitions.  Therefore,  the  various  work  centers  have  made  their  own  decisions 
regarding  these  past  performance  practices.  In  order  to  develop  a  policy  for  using  past 
performance  evaluations  we  must  first  understand  the  nature  of  implementing  past 
performance  practices  for  the  $1  million  to  $5  million  acquisition  range.  This  research 
study  examined  how  AFMC  work  centers  are  storing  and  protecting  their  past 
performance  information.  Additionally,  emphasis  was  placed  on  the  relevancy 
requirements,  information  sources,  and  information  types  that  are  useful  in  past 
performance  evaluations  for  acquisitions  below  $5  million.  Interviews  among  seven 
AFMC  work  centers  were  used  to  identify  common  and  dissimilar  characteristics  for  the 
$1  million  to  $5  million  acquisition  range. 

Specifically,  the  common  characteristics  among  these  seven  AFMC  work  centers 
indicate  peculiarities  for  the  $1  million  to  $5  million  acquisition  range.  The  approach  to 
storing  past  performance  information  is  divided  into  two  different  information  systems: 
Either  rely  on  predominant  use  of  questionnaires  or  maintain  an  “in-house”  database  for 
past  performance  information  below  $5  million.  Protection  methods  are  similar  to  that  of 
acquisitions  above  $5  million.  When  considering  relevancy  criteria  with  regards  to 
gathering  past  performance  information,  the  criteria.  Like  or  Similar  Service,  Key 
Individual,  Age  of  the  Information,  and  User’s  Input  are  the  four  major  criteria  prominent 
among  the  $1  million  to  $5  million  acquisition  range.  This  implies  that  objective 
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information  is  more  important  than  subjective  information  for  acquisitions  below  $5 
million.  This  research  effort  also  revealed  useful  information  sources  (e.  g.,  CPARS, 
questionnaires,  an  “in-house”  database,  the  Acquisition  Management  Information 
Service,  and  the  Contractor  Alert  List).  Additionally,  some  information  sources  are  not 
used  at  all  because  they  are  unknown  whereas  others  are  not  appropriate  for  this  range 
because  they  are  more  suitable  for  responsibility  determinations.  Finally,  the  most 
prominent  information  types  (Technical  Performance,  Unique  Aspects,  Quality 
Assurance,  and  Schedule  Control)  demonstrate  that  objective  information  is  the  most 
appropriate  for  the  $1  million  to  $5  million  acquisition  range. 

As  mentioned,  dissimilar  characteristics  were  found  among  the  AFMC  work 
centers  in  addition  to  the  commonalities.  Variance  exists  among  the  AFMC  work  centers 
in  their  treatment  of  past  performance.  What  is  similar  to  both  types  of  approaches  is  that 
past  performance  information  is  collected  and  considered  during  acquisitions  as  “Best 
Value  Practices”  echoes  the  moral  that  evaluations  should  not  always  be  based  on  price 
alone.  Without  the  regulatory  guidance,  the  various  work  centers  demonstrate  that  there 
is  more  than  one  way  to  skin  the  proverbial  cat.  Some  have  answered  with  the 
construction  of  a  database  to  store  their  collection  of  past  performance  information  while 
others  have  been  hesitant  for  justifiable  reasons.  From  this  central  difference  stems  other 
differences.  For  the  most  part,  the  regulations  for  acquisitions  above  $5  million  provide 
sufficient  guidance  for  acquisitions  under  $5  million.  However,  strict  compliance  to 
these  regulations  is  not  warranted  because  of  two  reasons:  1)  the  nature  of  these 
acquisitions  do  not  lend  the  feasibility;  and  2)  there  are  differences  in  approaches  found 
among  the  various  work  centers  because  what  might  be  useful  to  one  work  center  may 
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not  be  useful  to  another.  Surely,  the  AFMC  work  centers  can  gain  from  the  added  value 
of  knowing  how  other  AFMC  work  centers  are  conducting  past  performance  practices. 
But  standardization  is  not  the  means  of  providing  this  information  as  de-centralized 
decision  making  has  already  offered  them  the  opportunity  to  isolate  their  individual  needs 
and  respond  accordingly. 
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Appendix  A:  Interview  for  AFMC  Contracting  Personnel 


[Read  at  the  beginning  of  the  interview] 

We  are  conducting  a  research  study  for  HQ  AFMC.  The  study  is  for  the  purpose 
of  implementing  past  performance  guidelines  for  source  selections  between  $1,000,000 
and  $5,000,000. 

In  order  to  offer  us  a  meaningful  look  at  how  past  performance  information  has 
been  used,  your  anonymity  will  be  guaranteed,  if  you  prefer.  The  study  will  not  state 
which  AFMC  organizations,  nor  specific  programs,  were  used  in  obtaining  the  data. 

[for  interviews  not  conducted  via  telephone] 

This  interview  will  be  tape-recorded  upon  your  permission.  The  purpose  of  the 
tape  recorder  is  to  offer  more  reliable  transcription  of  the  data.  You  have  the  opportunity 
to  turn  off  the  tape  recorder  at  any  time  you  chose,  or  you  may  turn  off  the  recorder 
completely  for  the  duration  of  the  interview. 

[The  following  are  the  interview  questions] 


A.  Storing  the  Information. 


Question  #1: 
Question  #2: 
Question  #3: 

Question  #4: 
Question  #5: 
Question  #6: 
Question  #7: 
Question  #8: 

Question  #9: 


What  exactly  is  being  stored  in  your  past  performance  files? 
Is  the  information  stored  manually  or  in  electronic  form? 
How  often  is  past  performance  information  input  into  a 
contractor’s  file? 

How  often  is  the  information  updated? 

How  often  is  the  information  removed? 

How  are  contractor  name  changes  handled? 

How  are  contractor  mergers  handled? 

Is  there  a  program  manager  summary  similar  to  that  of  the 
CPAR  form? 

Are  there  other  actions  necessary  to  store  past  performance 
information? 


B.  Protecting  the  Information. 

Question  #10:  What  measures  are  done  to  ensure  proper  protection  of  the 
information? 

Question  #11:  Who  is  authorized  to  input  information  in  a  contractor’s 
file? 
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Question  #12:  Who  is  authorized  to  retrieve  information  from  a  file? 
Question  #13:  What  is  your  policy  towards  another  work  center  asking 
you  for  information  on  a  particular  contractor? 

Question  #14:  Does  a  contractor  have  the  opportunity  to  review  the  file? 


C.  Performing  Relevancy  Determinations. 

Question  #15:  For  what  kinds  of  requirements  are  you  keeping 
information? 

Question  #16:  What  criteria  do  you  use  in  performing  a  relevancy 
determination? 

Question  #17:  What  challenges  do  you  have  in  performing  relevancy 
determinations? 


D.  Sources  of  Past  Performance  Information. 


Answer  “Yes  or  No”  if  you  currently  consider  these  information  sources  to  obtain  past 
performance  information: 


Question  #18: 

Question  #19: 
Question  #20: 
Question  #21: 
Question  #22: 
Question  #23: 
Question  #24: 
Question  #25: 

Question  #26: 
Question  #27: 
Question  #28: 
Question  #29: 


Contractor  Performance  Assessment  Reporting  System 
(CPARS)  — for  contract  information  over  $5  million 
“In-house”  Performance  Reports 
Offerors’  Proposals 
Financial  Reports 

Contractor  Performance  Evaluation  Program  (CPEP) 
Acquisition  Management  Information  System  (AMIS) 
Legal  Databases  (such  as  LEXIS  or  WESTLAW) 
Contractor  Profile  System  (CPS)  or  the  Contractor 
Information  Service  (CIS) 

Contractor  Alert  List  (CAL) 

Other  Agency  Databases 
Questionnaires  and  Interviews 
Other  Sources 
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Answer  “Yes  or  No”  if  you  currently  consider  these  information  types  in  obtaining  past 
performance  information: 

Question  #30:  Cost  Control 

Question  #3 1 :  Timeliness  (Management  Responsiveness) 

Question  #32:  Timeliness  (Schedule  Control) 

Question  #33:  Business  Relations 

Question  #34:  Customer  Satisfaction  (Technical  Performance  of  Product) 
Question  #35:  Customer  Satisfaction  (Product  Assurance/Quality  System) 
Question  #36:  Customer  Satisfaction  (Product  Assurance/Manufacturing 
Management) 

Question  #37:  Customer  Satisfaction  (Logistics  Support/Sustainment) 
Question  #38:  Customer  Satisfaction  (Engineering) 

Question  #39:  Key  Personnel  (Program/Data  Management) 

Question  #40:  Key  Personnel  (Procurement/Subcontract  Management) 

Question  #41 :  Unique  Aspects 

Question  #42:  Other  Aspects — not  already  mentioned 
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